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FOREWORD 


The  work  accomplished  in  the  "FLIR  Sensor  Processor"  program,  results  of  which 
are  contained  in  this  report,  promises  to  provide  significant  benefits  in  the 
operational  effectiveness  of  future  FLIR  sensor  systems. 

Northrop  Corporation,  Electro-Mechanical  Division  wishes  to  express  appreciation 
to  those  U.S.  Government  organizations  who  sponsored  and  actively  supported  this 
important  technical  activity.  Sponsorship  for  this  program  was  provided  ty  the 
Department  of  the  Army's  Advanced  Concepts  Team,  headed  by  Charles  H.  Church, 
under  the  Deputy  Chief  of  Staff  for  Research,  Development,  and  Acquisition.  The 
program  was  managed  by  the  U.S.  Army  Electronics  Command's  Night  Vision  Labora- 
tory. Exceptional  technical  guidance  was  provided  by  John  S.  Dehne,  who  heads 
the  Image  Processing  Group,  and  G.  David  Singer,  Project  Engineer. 


ill 


Section 


CONTENTS  * 


INTRODUCTION 


1.1  Quarterly  Review  

1.2  Report  Outline  

1.3  Image  Processing  Facility.  . . . 

1.4  Fast  Rationalized  Haar  Transform 

1.4.1  Basic  Definitions 

1.4.2  Two-Dimensional  Formats 

1.4.3  Matrix  Representation 


FOCUS  EXPERIMENTS. 


2.1  introduction  

2.2  General  Experimental  Procedure  . . 

2.3  / Test  Pattern  Experiment 

2.3.1  Candidate  Focus  Measures  

2.3.2  Focus  Measure  Data  

2.3.3  Summary 

2.4  FLIR  Image  Experiment 

2.4.1  Image  Data  

2.4.2  Summary  of  FLIR  Image  Experiments. 

2.5  Conclusions 


DEBLUR  EXPERIMENTS. 


3.1  One  Dimensional  Blur  Model  . . . . 

3.2  Blur  Operations 

3.2.1  Focus  Error 

3.2.2  Diffraction  Limited  Aperture  . . . 

3.2.3  Translation 

3.2.4  Summary 

3.3  A Diagonal  Representation  of  Blur. 

3.3.1  Focus  Error 

3.3.2  Limited  Aperture  

3.3.3  Image  Motion  

3.4  Approximate  Diagonal  Deblur.  . . . 

3.5  Motion  Detection/Correction.  . . . 

3.6  Two  Dimensional  Blur  Model  . . . . 

3.7  Test  Images  

3.8  Conclusion  


CONTENTS  (Continued) 


> EDGE 

4.1 

4.2 

4.3 
4.3. 
4.3. 
4.4/ 
4*4 


7 

4^.4. 

/4 . 4 . 

4.5 

4.6 


ENHANCEMENT  EXPERIMENTS 

Introduction  

General  Experimental  Procedure 
Edge  Enhancement  Criteria.  . . 

Definition  of  Edges 

Weighting  Factors 

Edge  Enhancement  Experiment.  . 

Constant  Multiplier 

Variable  Multiplier 

Descriptors 

FLIR  Image  Experiments  .... 
Conclusions 


^ ADAPTIVE  SIGNAL  PROCESSING 

Introduction  

DC  Restoration  Problem  

Problem  Definition  

DC  Restoration  Solution 

Dynamic  Range  Compression 

Problem  Definition  

Dynamic  Range  Compression  Solution 
Conclusions 


BANDWIDTH  COMPRESSION r 
* > 


6.1  Introduction  

6.2  General  Experimental  Procedure  

6.3  Reduced  Resolution  Compression  

6.3.1  Scan  Line  Sampling  

6.3.2  Area  Sampling 

6.3.3  Intensity  Sampling  

6.3.4  Reduced  Resolution  Summary  

6.4  Haar  Transform  Compression  

6.4.1  Sequency  Domain  Bit  Assignments 

6.4.2  Coefficient  Elimination 

6.5  Image  Descriptors  for  Haar-Compressed  Images 

6.6  Summary  and  Conclusions 


IMAGE  DESCRIPTORS. 


} ■ — 

7.1  Introduction  

7.2  Image  Description  in  the  Haar  Domain 

7.2.1  Local  Image  Descriptors 

7.2.2  Global  Image  Descriptors  

7.3  Descriptor  Results  

7.3.1  Focus  Results 

7.3.2  Enhancement  Results 


Page 

4-1 

4-1 

4-2 

4-4 

4-4 

4-11 

4-16 

4-16 

4-20 

4-21 

4-23 

4- 34 

5- 1 

5-1 

5-1 

5-1 

5-5 

5-23 

5-23 

5-24 

5- 26 

6- 1 

6-1 

6-1 

6-3 

6-3 

6-6 

6-6 

6-6 

6-9 

6-9 

6-9 

6-21 

6- 29 

7- 1 

7-1 

7-2 

7-2 

7-9 

7-13 

7-14 

7-14 


1 


V 


Section 


CONTENTS  (Continued) 


Page 


7.3.3  Compression  Results 7-15 

7.4  Conclusions  and  Recommendations 7-16 


Addendum 


SENSOR  PROCESSOR  UNIT  - DEMONSTRATION  PROTOTYPE.  . . 

) 

8.1  System  Description,  Sensor  Processing  Unit  . 

8.2  System  Block  Diagram  

8.3  Hardware  Configuration  

^ AUTOFOCUS  DEMONSTRATION*  ^ 


8-1 

8-1 

8-2 

8-4 

A- 1 


Figure 


ILLUSTRATIONS 


1-1  Image-Processing  Facility 1-3 

1-2  General  Haar  Function  h(r,m,x)  1-5 

1-3  General  Two-Dimensional  Rationalized  Haar  Production 

Function 1-6 

1-4  Haar  Image  Domain 1-6 

1-5  Two-Dimensional  Rationalized  Haar  Functions 1-7 

1- 6  Format  of  Northrop's  Real-Time  2-D  Haar  Processor 1-10 

2- 1  Experimental  Test  Configuration 2-3 

2-2  Test  Imagery 2-4 

2-2  Autofocus  Test  Setup 2-5 

2-4  Portion  of  Lens  Cycle  Printout 2-8 

2-5  Test  Pattern  Images  as  a Function  of  Lens  Position  2-10 

2-6  S , Sum  of  AC  Magnitudes,  as  a Function  of  Lens  Position  . . 2-13 

2-7  S , Sum  of  Normalized  AC  Magnitudes,  as  a Function  of 

Lens  Position 2-14 

2-8  C,  Sum  of  AC  Power,  as  a Function  of  Lens  Position  2-15 

2-9  RMS  Noise  Levels  for  S , Sum  of  AC  Magnitudes 2-16 

2-10  RMS  Noise  Levels  for  Sm,  Sum  of  Normalized  AC  Magnitudes  . . 2-17 

2-11  RMS  Noise  Levelc  for  C?  Sum  of  AC  Power 2-18 

2-12  A Comparison  of  Focus  Measures 2-19 

2-13  Test  Pattern  Descriptor  Input  Areas 2-20 

2-14  Activity  Descriptors  as  a Function  of  Lens  Position 2-20 

2-15  FLIR  Images  with  Input  Areas  Flagged  by  Increased 

Brightness 2-24 

2-16  Total  AC  Power  - Image  No.  1 2-25 

2-17  Total  AC  Power  - Image  No.  2 2-26 

2-18  Total  AC  Power  - Image  No.  3 2-27 

2-19  Total  AC  Power  - Image  No.  4 2-28 

2-20  Total  AC  Power  - Image  No.  5 2-29 

2-21  Total  AC  Power  - Image  No.  5 (Alternate  Sectors) 2-30 

2-22  Total  AC  Power  - Image  No.  1 Target 2-32 

2-23  Total  AC  Power  - Image  No.  2 Target 2-32 

2-24  Total  AC  Power  - Image  No.  3 Target 2-33 

2-25  Total  AC  Power  - Image  No.  4 Target 2-33 

2-26  Total  AC  Power  - Image  No.  5 Target 2-34 


ILLUSTRATIONS  (Continued) 


Figure  Page 


2-27  Activity  Descriptors  as  a Function  of  Lens  Position  - 

Image  No.  1 Background 2-34 

2-28  Activity  Descriptors  as  a Function  of  Lens  Position  - 

Image  No.  1 Target 2-35 

2-29  Activity  Descriptors  as  a Function  of  Lens  Position  - 

Image  No.  4 Background 2-35 

2- 30  Activity  Descriptors  as  a Function  of  Lens  Position  - 

Image  No.  4 Target 2-36 

3- 1  Impulse  Responses  of  Focus  Error  3-4 

3-2  Haar  Domain  Operator  Focus  Error  (Case  1) 3-5 

3-3  Haar  Domain  Operator  Focus  Error  (Case  2) 3-6 

3-4  Haar  Domain  Operator  Aperture  Blur  (Case  1) 3-8 

3-5  Haar  Domain  Operator  Aperture  Blur  (Case  2) 3-9 

3-6  Haar  Domain  Operator  One-Pel  Translation  3-10 

3-7  Haar  Domain  Operator  Two-Pel  Translation  3-11 

3-8  Sequency  Attenuation  as  a Result  of  Focus  Error 3-14 

3-9  Sequency  Attenuation  as  a Result  of  Limited  Aperture  ....  3-15 

3-10  FLIR  Thermograms 3-19 

3-11  Average  Magnitude  Haar  Spectra  3-20 

3-12  Haar  Spectrum  Prototype 3-21 

3-13  Haar  Domain  Operator  for  a One-Pel  Left  Shift 3-22 

3-14  Haar  Domain  Operator  for  a One-Pel  Right  Shift  3-23 

3-15  Motion  Detection  Method  3-24 

3-16  Deblur  Results  for  Image  #1 3-28 

3-17  Deblur  Results  for  Image  #5 3-28 

3-18  Deblur  Results  for  Image  #6 3-29 

3-19  Deblur  Results  for  Image  #7 3-29 

3-20  Deblur  Results  for  Image  #8 3-30 

3- 21  Deblur  Results  for  Image  #10 3-30 

4- 1  Experimental  Test  Configuration 4-3 

4-2  FLIR  Thermograms 4-5 

4-3  Haar  Functions,  Groups  4 through  15 4-7 

4-4  All  AC  Coefficients,  Image  1 4-9 

4-5  Groups  9,  11,  and  12,  Image  1 4-9 

4-6  Groups  10  and  13,  Image  1 4-10 

4-7  Group  14,  Image  1 4-10 

4-8  Group  15,  Image  1 4-11 

4-9  All  AC  Coefficients 4-12 

4-10  AC  Coefficients  for  all  Six  Test  Images  (Composite) 4-13 

4-11  Groups  9 through  15  with  Identical  Amplitude  4-14 

4-12  Group  Normalization 4-15 

4-13  Effects  of  Different  Edge  Thresholds  4-17 

4-14  Effects  of  Different  Multipliers  4-18 

4-15  Effects  of  System  Noise 4-19 

4-16  Effects  of  TV  2:1  Interface 1 4-20 

4-17  Nonlinear  Transfer  Functions  4-22 

4-18  Variable  Multiplier  Enhancement  - Transfer  Function  No.  1.  . 4-23 

4-19  Variable  Multiplier  Enhancement  - Transfer  Function  No.  2.  . 4-24 

4-20  Image  1 4-26 

4-21  Image  5 4-27 


vii 


f- 


5-9 
5-10 
- 5-11 
5-12 

5- 13 

6- 1 
6-2 
6-3 
6-4 
6-5 
6-6 
6-7 
6-8 
6-9 
6-10 
6-11 

6-12 

6-13 

6-14 

6-15 

6-16 

6-17 

6-18 

6- 19 

7- 1 
7-2 
7-3 
7-4 
7-5 
7-6 

7- 7 

8- 1 
8-2 


ILLUSTRATIONS  (Continued) 


Image  6 4-28 

Image  7 4-29 

Image  8 4-30 

Image  10 4-31 

Parking  Lot  Scene 4-33 

AC  Coupling  Effects  in  Parallel  Scan  FLIRS  5-2 

Vertical  Correlation  Loss  in  Parallel  Scan  FLIRS  5-3 

Ideal  DC-  and  AC-Coupled  Infrared  Imaging 5-4 

Weighting  Function  of  AC  Coupling  Circuit 5-4 

Multiscan  Droop  Characterization  5-6 

Multiscan  Droop  Correction  Analog 5-8 

Conservative  Representation  of  Parallel-Scan  FLIR  High-Pass 

Filter 5-10 

Processor  Algorithm  for  Absolutely  Stable  Corrective 

Network 5-11 

Gray-Scale  Centering  Algorithm  5-18 

Non-Artifacting  Brightness  Control  Scheme 5-20 

Contrast  Linearization  Analytical  Model 5-21 

Haar  Contrast  Control 5-22 

Dynamic  Range  Compression  Example 5-25 

Experimental  Test  Configuration 6-2 

Transform  Format  6-4 

Under  Sampled  Scan  Lines 6-5 

Under  Sampled  Areas 6-7 

Reduced  Quantization  Levels 6-8 

Example  of  Amplitude  Distributions  6-10 

Test  Pattern  Compressed  by  Sequency  Domain  BIT  Limiting.  . . 6-11 

FLIR  Image  Compressed  by  Sequency  Domain  BIT  Limiting.  . . . 6-12 

Zonal  Filtered  Test  Pattern 6-14 

Zonal  Filtered  FLIR  No.  1 6-15 

Effects  of  Threshold  Levels  in  Sequency  Domain, 

Test  Pattern 6-17 

Effects  of  Threshold  Levels  in  Sequency  Domain, 

FLIR  No.  1 6-18 

Threshold  and  N-Coef f icients  Test  Pattern 6-19 

Threshold  and  N-Coef f icients  (FLIR  No.  1) 6-20 

Threshold  and  N-Coef f icients  (NVL  No.  4) 6-22 

Threshold  and  N-Coef f icients  (NVL  No.  6) 6-23 

Threshold  and  N-Coef f icients  (NVL  No.  7) 6-24 

Threshold  and  N-Coef f icients  (NVL  No.  8) 6-25 

Threshold  and  N-Coef f icients  (NVL  No.  10) 6-26 

Group  Edge  Weighting,  a rp 7-5 

Representative  Histograms  of  Coefficients  of  j Group  . . . 7-6 

Sample  Images 7-7 

Examples  of  Local  Tone  and  Activity 7-8 

Examples  of  Local  Texture  and  Edge  Activity 7-9 

Histogram  of  Local  Tone  of  Girl 7-11 

Histogram  of  Local  Tone  of  Moon 7-11 

Top-Level  Functional  Block  Diagram  8-3 

Hardware  Configuration  8-5 


J 

a 

a 


viii 


li 

ft 


TABLES 


Table  Page 


2-1  Focus  Measure  Signal-to-Noise  Ratios  2-19 

2-2  Focus  Measure  Computational  Complexity  2-21 

2- 3  Performance  of  Candidate  Focus  Measures 2-22 

3- 1  Grouping  of  the  HaarDomain  Operator  3-3 

3-2  MSE  Observed  Using  D to  Deblur 3-17 

3- 3  Summary  of  Deblur  Results  3-31 

4- 1  Descriptors  for  Test  Images 4-6 

4-2  Average  Adaptive  Threshold  (Image  1)  4-7 

4-3  Percent  Edge  Terms 4-8 

4-4  Number  of  Terms  Exceeding  ©.(Image  1) 4-8 

4-5  Selected  Parameters  for  Constant  Multiplier 4-20 

4-6  Descriptors  for  Image  1 4-25 

4- 7  Descriptors  of  Selected  Enhancement  Algorithms  4-32 

5- 1  Computer  Simulations  for  Matched  and  Mismatched  AC  Coupling 

Corrective  Networks . 5-14 

5- 2  Rationalized  Haar  Components  of  g(x) 5-26 

6- 1  Descriptors  for  Haar-Compressed  Images  6-27 

7- 1  Statistics  of  Local  Tone  and  Activity 7-10 

7-2  Statistics  of  Local  Edge  and  Texture  Activity 7-13 


SECTION  1 
INTRODUCTION 


Northrop  Corporation,  Electro-Mechanical  Division,  is  pleased  to  submit  the  final 
report  of  the  Experimental  Development  of  a FLIR  Sensor  Processor.  The  program's 
funding  started  on  July  1,  1976,  and  this  report  summarizes  the  technical  perform- 
ance through  September  30,  1977,  marking  the  completion  of  Contract  No.  DAAG53- 
C-0188 . 

1.1  QUARTERLY  REVIEW 

During  the  first  quarter,  the  Experiment  and  Study  Plan  was  written,  the  test  sys- 
tem configuration  was  assembled  and  the  software  was  debugged.  In  addition  to  the 
Experiment  and  Study  Plan,  the  first  quarterly  report  included  a brief  Advanced 
Scout  Helicopter  mission  scenario  used  to  characterize  the  FLIR  operational  environ- 
ment. The  report  also  included  basic  definitions  of  the  Haar  functions  used  in 
the  experiments  and  a set  of  image  descriptors,  designed  to  facilitate  evaluation 
of  the  image  processing  algorithms. 


The  second  quarterly  report  primarily  covered  the  autofocus  experiments.  The 
object  was  to  determine  the  use  of  the  2D  Haar  processor  as  an  accurate  sensor  of 
infocus/defocus  conditions  in  the  FLIR  systems.  The  results  of  the  experiments 
and  subsequent  analysis  showed  that  using  FLIR  imagery,  with  some  amount  of  acti- 
vity within  the  view,  a simple  scalar  value  can  be  computed  in  real  time,  which 
has  a sensitivity  greater  thai  the  eye  in  determining  best  focus. 


The  third  quarterly  report  summarized  the  experiments  performed  to  determine  how 
the  2D  Haar  processor  can  be  used  to  enhance  FLIR  imagery  by  highlighting  edge 
details.  Algorithms  were  developed  to  extract  edge  information  and  then  to  enhance 
the  image.  These  algorithms  were  tested  on  several  thermograms  provided  by  Night 
Vision  Laboratory.  Results  were  analyzed  and  the  imagery  included.  A fixed  edge 
threshold  algorithm  was  reommended  for  the  extraction  of  edges  and  a variable 
multiplier  algorithm  was  recommended  for  edge  emphasis. 
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1.2  REPORT  OUTLINE 

For  the  convenience  of  the  reader,  this  final  report  includes  much  of  the  material 
which  previously  appeared  in  the  three  quarterly  reports.  An  outline  of  the  final 
report  follows. 

Section  1,  Introduction,  includes  a description  of  the  image  processing  facility 
and  a discussion  of  the  Fast-Rationalized  Haar  transform,  which  forms  the  basis 
for  all  the  image  processing  presented  in  thi3  report. 

Section  2 contains  the  autofocus  experiments. 

Section  3 describes  the  deblur  experiments.  This  includes  blurs  caused  by  defocus- 
ing,  diffraction  limitations  and  scene  translations.  A method  of  formulating  these 
blur  characteristics  in  the  Haar  domain  and  operations  leading  to  simple  deblur- 
ring algorithms  are  presented. 

Section  4 consists  of  the  edge  enhancement  experiments. 

Section  5 includes  the  experiments  performed  in  DC  restoration  dynamic  range  com- 
pression and,  gain  and  brightness  control  under  the  general  heading  "Adaptive  Signal 
Processing" . 

Section  6 presents  the  data  compression  experiments.  Several  techniques  that  reduce 
image  encoding  data  rates  by  removing  redundancy  are  described  and  compared  with 
filtering  techniques  that  reduce  resolution.  The  processed  pictures  and  image 
statistics  are  included. 

Section  7 provides  a recap  of  the  Image  Descriptors  used  to  evaluate  Image  quality. 

Section  8 contains  a suggested  hardware  implementation  of  some  of  the  most  success- 
ful algorithms. 

The  results  pf  the  autofocus  demonstration,  held  at  Wright-Patterson  Air  Force 
Base  in  September  1977, 'will  be  presented  as  an  addendum  to  this  report.  Here, 
the  2-D  Haar  processor  was  coupled  to  an  operating  FLIR  to  demonstrate  its  real- 
time autofocus  capabilities. 
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sensor  imagery  and  of  digitizing  and  converting  the  images  into  an  equivalent  ensem- 
ble of  Haar  transform  coefficients.  Real-time  and  off-line  processing  can  be  per- 
formed on  these  coefficients  prior  to  transformation  back  to  images.  Image  and 
transform  statistics  are  assembled,  computed,  and  printed  out  for  further  analysis.  ^ 

1.4  FAST  RATIONALIZED  HAAR  TRANSFORM 

The  following  paragraph  covers  the  basic  definition  of  Haar  functions,  their  exten- 
sion to  two-dimensional  formats  for  image  processing  applications,  and  their 
expression  in  terms  of  matrices. 

1.4.1  Basic  Definitions 

Haar  transforms  provide  the  basis  for  all  image-processing  techniques  presented 
in  this  report.  Like  other  transforms,  they  can  assign  quantitative  measures  to 
qualitative  aspects  of  the  original  image,  or  replace  an  involved  measurement  tech- 
nique by  an  easier  and  faster  one.  These  measures,  in  turn,  identify  the  important 
variables  for  processing  image  restoration  and  enhancement. 

Haar  transforms  are  also  comparatively  simple.  Each  transform  component  effectively 
multiplies  the  original  function  by  either  0,  +1,  or  -1.  Furthermore,  they  combine 
the  abilities  of  local  transforms  (e.g.,  Block  transforms)  to  touch  up  given  areas 
of  a picture,  with  those  of  global  transforms  (e.g.,  Walsh  transforms)  to  render 
broad  comprehensive  interpretation  of  data. 

The  general  Haar  function  h(r,m,x)  appears  in  Figure  1.2.  Sequency  r bears  a cer- 

tain  resemblance  to  frequency.  Offset  m ranges  between  1 and  2 . These  mutually 

r / 2 

orthogonal  functions  can  be  rationalized  by  reducing  all  amplitudes  2 to  a 
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1.4.2  Two-Dimensional  Formats 

Haar  functions  can  be  used  to  expand  general  one-dimensional  (ID)  waveforms.  Two 
dimensional  (2D)  waveforms,  such  as  image  intensity  distributions  l(x,y)  can  simi- 
larly be  expanded  in  a product  series: 

I(x,y)  = 2 armpn  h(r ,m,x)h(p,n,y)  (1) 

r ,m,p,n 

1/2  1/2 

armpn  ~ j f Kx>y)h(r  ,m,x)h(p,n,y)dxdy  (2) 

-1/2  -1/2 

While  this  applies  to  preliminary  domain  -1/2  s x s 1/2,  -1/2  <y  s 1/2,  simple 
linear  scale  changes  will  qualify  these  relations  for  images  of  any  finite  extent. 

A rationalized  two-dimensional  product  h(r ,n,x)h(p,n,y)  appears  as  in  Figure  1-3. 

When  all  such  diagrams  have  been  located  in  rectangular  (r,p)  coordinate  space 
as  in  Figure  1-4,  the  resulting  arrey  represents  the  Haar  image  transform  domain. 

A detailed  picture  of  the  array  is  given  in  Figure  1-5.  It  is  possible  to  segment 
this  domain  into  regions  in  which  the  Haar  functions  are  simply  translations  of 
one  another.  Each  of  the  largest  possible  regions  of  the  Haar  domain  in  which 
the  Haar  functions  exhibit  this  property  is  referred  to  as  a sequency  group.  The 
set  of  16  x 16  Haar  basis  functions  can  be  segmented  into  25  sequency  groups. 
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Examination  of  Figure  1-5  shows  that,  by  grouping  the  Haar  coefficients  as  shown,  it 
is  possible  to  separate  those  coefficients  corresponding  to  the  horizontal  or 
vertical  edges  in  the  image.  Assuming  that  horizontal  and  vertical  edges  are 
equally  likely  in  a typical  image,  these  25  groups  can  be  combined  to  get  15  new 
groups  as  shown  by  the  numbering  scheme  in  Figure  1-5  and  as  enumerated  below: 

Group  No.  of  Terms  in  Group 


Sum  = 256 

With  the  Northrop  Haar  processor,  it  is  possible  to  select  lowrr  and  upper  thresh- 
olds and  a multiplication  factor  independently  for  each  group. 

1.4.3  Matrix  Representation 

Forward  and  inverse  Haar  transforms  in  equations  (1)  and  (2)  can  also  be  expressed 
in  terms  of  matrices.  It  should  be  noted  that  matrix  expressions  substitute  dis- 
crete summation  for  integration  in  equation  (2).  With  this  provision,  continuous 
image  distribution  I(x,y)  reduces  to  the  discrete  expression  I(t4x,  lAy)  where 
4x  and  Ay  typically  assume  values  smaller  than  the  resolution  spot  size  of  the 
imaging  system.  Integers  t and  1 span  a range  sufficient  to  cover  maximum  and 
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minimum  values  of  x and  y.  This  expression  in  turn  can  be  represented  by  the  matrix 

I . or  simply  I. 

C , 1 

Haar  function  h(r,m,x)  reduces  to  the  matrix  h where  integer  s designates  succes- 

L S 

sive  Haar  functions  in  a particular  sequency  and  then  moves  on  to  the  next  sequency. 
Using  only  the  first  few  sequencies  in  a rationalized  format  to  clarify  the  illus- 
tration: 


1 1 


1111 


1 1 1 1 -1  -1  -1  -1 


1 1-1-1 


0 0 0 0 
1 1-1-1 


rationalized 


1 -1 


* M 


rationalized 


1 -1  0 


1 -1 


Expansion  coefficient  a likewise  reduces  to  matrix  a = 
r rmpn  [ sq  j 

Equations  (1)  and  (2)  then  convert  to  the  matrix  expressions 


Because  of  the  simplicity  of  (3)  and  the  compactness  of  (4)  and  (5),  the  matrix 
approach  will  be  preferred  in  general. 

Northrop' s real-time  Haar  processor  performs  the  rationalized  Haar  transform  on  16 
x 16  pel  non-overlapping  blocks  of  a digitized  image  array.  The  precise  arrangement 
of  transform  blocks  and  data  processing  rat6s  is  depicted  in  Figure  1-6. 


SECTION  2 

AUTO  FOCUS  EXPERIMENTS 


2 . 1 INTRODUCTION 

Because  severe  demands  are  placed  on  a two-man  helicopter  team,  the  need  for  manual 
FLIR  refocusing  can  pose  a formidable  problem.  FLIR  defocusing  arises  from  continual 
changes  in  observation  range,  small  thermal  deformations  of  the  lens,  and  other 
incidental  causes.  Automatic  refocusing  offers  a means  to  combat  these  tendencies 
and  relieve  part  of  the  manpower  burden.  The  apparatus  must  sense  the  defocus. con- 
dition of  the  image  and  produce  a corresponding  error  signal  for  automatic  lens  ad- 
justment. The  objective  of  this  experiment  is  to  develop  a measure  of  the  state  of 
focus  of  an  image  based  on  the  distribution  of  energy  in  its  rationalized  Haar 
transform.  The  resultant  focus  measure  may  then  be  used  as  the  basis  for  the  gen- 
eration of  a focus-correcting  error  signal. 

The  focus  measure  developed  should  exhibit  the  following  characteristics  in  order 
to  be  compatible  with  system  and  operational  requirements: 

a.  Low  Computational  Complexity  - The  amount  of  computation  required  to 
evaluate  an  image's  state  of  focus  must  be  amenable  to  real-time  (TV  rates) 
processing. 

b.  High  Sensitivity  to  Focus  - Small  changes  in  an  image's  state  of  focus 
should  produce  significant  changes  in  the  focus  measure. 

c.  Noise  Immunity  - In  the  present  test  configuration,  noise  in  the  video 
signal  before  digitization  is  the  overriding  contributor  to  system  noise.  In  an 
operational  configuration,  small  (one  pel)  translations  of  the  image  may  also  pro- 
duce a significant  amount  of  noise.  The  focus  measure  should  be  relatively  insen- 
sitive to  both  types  of  image  noise. 


2-1 


d.  Scene  Independence  - The  focus  measure  should  perform  equally  well  on 
both  background  and  targets;  however,  there  are  some  limitations  to  this  requirement. 
There  must  be  sufficient  activity  in  the  area  of  interest  to  evaluate  the  image's 
state  of  focus.  For  example,  any  focus  measure  will  not  be  expected  to  perform  well 
when  the  image  contains  only  clear  sky. 

e.  Adaptability  in  an  Operational  System  - The  focus  measure  in  its  final 
form  must  be  able  to  serve  as  the  basis  for  the  generation  of  an  error  signal  in  a 
servo  system.  This  signal  must  have  sufficient  bandwidth  and  stability  to  be  com- 
patible with  FLIR  image  quality  parameters  and  ASH/AAH  operational  requirements. 

These  specific  system  requirements  are  discussed  in  the  Appendix  to  the  first 
quarterly  report  (76Y133) . 

The  experiment  is  divided  into  two  major  sections.  In  the  first  section  the  per- 
formance of  several  candidate  focus  measures  is  examined.  The  test  image  used  in 
this  section  is  a video  test  pattern  (RETMA  resolution  chart).  Tradeoffs  of  the 
characteristics  of  the  candidate  focus  measures  are  made  and  two  of  them  are  selected 
for  subsequent  analysis.  In  the  second  section,  the  capabilities  of  the  selected 
focus  measures  are  evaluated  on  the  set  of  five  FLIR  thermograms  supplied  by  NVL. 

Both  target  and  background  objects  are  used  as  input  data  in  order  to  determine  the 
behavior  of  the  focus  measures  on  general  classes  of  images. 


2.2  GENERAL  EXPERIMENTAL  PROCEDURE 


The  experimental  test  configuration  is  shown  in  Figure  2-1.  Test  imagery  consisted 
of  a video  test  pattern  and  five  FLIR  thermograms  supplied  by  Night  Vision  Labora- 
tory (NVL)  (Figure  2-2).  The  focus  of  the  video  camera  was  set  manually  with  the  aid 
of  13  equally  spaced  marks  on  the  lens  barrel.  Figure  2-3  shows  the  optical  rail 
apparatus  and  a more  detailed  view  of  the  video  camera  lens. 

The  data  collection  sequence  for  each  of  the  focus  measures  was  as  follows:  The 
video  camera  lens  was  set  initially  at  one  extreme  (lens  position  A).  The  Haar 
transforms  of  both  fields  of  four  previously  selected  sectors  of  the  image  were 
acquired  and  the  candidate  focus  measure  computed.  The  result  was  then  printed  out 
on  the  Centronics  Line  Printer. 


2-2 


Figure  2-1.  EXPERIMENTAL  TEST  CONFIGURATION 


la)  VIDEO  TEST  PATTERN 


(c)  IMAGE  2 


Id)  IMAGE  3 


la)  IMAGE  4 


If)  IMAGE  5 


Figure  2-2.  TEST  IMAGERY 


AUTOFOCUS  TEST  SETUP 


This  was  repeated,  advancing  the  lens  position  one  step  each  time,  until  the  other 
extreme  lens  position  (lens  position  M)  was  reached.  The  entire  process  was  then 
repeated,  stepping  the  lens  backward  from  position  M to  position  A.  This  process 
of  advancing  and  retracting  the  lens  position  through  one  cycle  while  evaluating  a 
focus  measure  at  each  position  will  be  referred  to  as  a "Lens  Cycle."  One  lens 
cycle  takes  about  6 minutes  to  complete. 

Noise  measurements  of  each  focus  measure  were  performed  by  constantly  computing  the 
focus  measure's  value  while  leaving  the  video  camera  lens  stationary. 

2.3  TEST  PATTERN  EXPERIMENT 

2.3.1  Candidate  Focus  Measures 


Several  preliminary  measures  of  an  image's  state  of  focus  were  selected  for  study. 
Each  measure  is  based  on  the  distribution  of  energy  in  the  Haar  transform  of  the 
image.  The  candidate  focus  measures  are  presented  here  in  order  of  increasing  com- 
putational complexity. 


2. 3. 1.1  Sum  of  the  Magnitudes  - The  sum  of  the  magnitudes,  S^,  of  the  ac  Haar 
coefficients  is  expressed  as 


S 

m 


l HI  \a  | 

L Li  L L \ rmpn  1 

r m p n 


a-i,i,-i,i 


(i) 


S is  the  computationally  simplest  index  of  the  total  ac  information  in  an  image's 
m 

Haar  transform.  In  order  to  ascertain  the  distribution  of  ac  information  through- 
out the  transform  as  well,  totals  of  the  magnitudes  of  the  coefficients  were  also 
taken  for  each  sequency  group. 


2. 3.1.2  Sum  of  the  Normalized  Magnitudes 

S , of  the  ac  coefficients  is  expressed  as 
n 


The  sum  of  the  normalized  magnitudes, 


HU  f | a | 
L L L L rp  1 rmpn1 
rmpn 


-1,1, -1,1 


(2) 
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The  two-dimensional  sensor  processor's  implementation  of  the  rationalized  Haar 

transform  contains  an  inherent  high  sequency  boost.  To  remove  this  boost,  a nor- 

til 

malization  factor,  f , is  introduced  to  weight  the  information  from  the  rp 

sequency  group.  The  sum  of  the  normalized  magnitudes  more  closely  approximates 

the  sum  of  the  magnitudes  of  an  unrationalized  Haar  transform.  S was  evaluated 

m 

for  all  the  ac  terms  as  well  as  for  each  sequency  group. 


2. 3. 1.3  AC  Power  - The  ac  power,  C,  is  defined  as 

C - I H l (^n)2  - <a  >2  (3) 

r m p n rapn  -1*1*-1’1 

The  parameter  C is  equivalent  to  the  local  activity  of  a 16  by  16  pel  image  sector. 
The  image's  activity  should  provide  an  index  of  focus  that  is  more  noise  immune  and 
more  sensitive  to  focus  than  the  previous  measures.  The  noise  immunity  arises  from 
the  fact  that,  by  squaring  terms,  a few  large  fluctuations  in  amplitude  may  still 
be  seen  despite  the  presence  of  many  small  ones.  The  increased  sensitivity  to 
focus  is  due  to  a similarity  between  the  Fourier  and  unrationalized  Haar  transforms. 
Since  both  transforms  are  unitary  and  possess  equivalent  dc  terms,  Parseval's  theorem 
tells  us  that  the  ac  energy  in  each  is  also  equivalent.  Defocusing  an  image  effec- 
tively subjects  the  image  to  a low-pass  filter,  thereby  decreasing  the  ac  energy  in 
its  Fourier  transform.  The  ac  energy  in  the  image's  unrationalized  Haar  transform 
would  likewise  be  decreased.  Since  the  rationalization  of  the  Haar  transform  in- 
volves only  simple  scaling  of  each  sequency  group,  the  direction,  if  not  the  amount, 
of  change  in  the  ac  energy  of  the  rationalized  Haar  will  be  equivalent  to  that  of 
the  unrationalized  Haar  and  Fourier  transforms. 


I 


2. 3. 1.4  Texture  and  Edge  Activity  - Texture  and  edge  activities,  as  defined  in 
the  first  quarter  report,  are  the  most  expensive  computationally.  They  are  included 
in  the  study  to  determine  if  variations  in  image  activity  due  to  changes  in  state 
of  focus  will  appear  primarily  as  variations  in  texture  or  edge  activity. 


2.3.2  Focus  Measure  Data 

Each  of  the  foregoing  focus  measures  was  computed  through  one  lens  cycle.  A repre- 
sentative portion  of  a lens  cycle  printout  is  shown  in  Figure  2-4.  Since  during 
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each  lens  cycle  each  lens  position  is  encountered  twice,  the  two  resultant  focus 
measure  values  are  averaged  to  facilitate  graphical  presentation  of  the  data. 

The  range  of  defocus  encountered  through  one  lens  cycle  is  shown  in  the  series  of 
photographs  in  Figure  2-5,  A through  M.  The  behavior  of  the  first  three  candidate 
focus  measures  is  shown  as  a function  of  lens  position  in  Figures  2-6,  2-7,  and  2-8. 
The  noise  characteristics  of  each  measure  are  shown  in  Figures  2-9,  2-10,  and  2-11. 

As  can  be  seen  in  Figures  2-6,  2-7,  and  2-8,  each  focus  measure  tested  exhibited  a 
peak  value  at  lens  position  G.  In  order  to  determine  if  lens  position  G is  indeed 
the  sharpest  focus  position,  the  fine  detail  of  the  test  pattern  was  observed  under  a 
hand  lens.  Close  examination  of  the  wedges  in  the  test  pattern  (Figure  2-5)  shows 
that  lens  positions  F,  G,  and  H are  the  ones  that  yield  images  with  sharper  focus. 
Visually  discriminating  the  difference  in  focus  between  F,  G,  and  H is  next  to  im- 
possible, however.  Since  lens  position  G is  in  the  center  of  F,  G,  and  H,  and  the 
peaks  in  the  focus  measure  curves  are  distinct,  lens  position  G is  deemed  to  be  the 
"in  focus"  lens  position.  The  ability  of  the  focus  measure  to  evaluate  the  state  of 
focus  of  an  image  appears  to  rival  that  of  the  observer. 

The  division  of  energy  into  sequency  groupings  for  each  focus  measure  can  be  seen 
in  part  A of  Figures  2-6,  2-7,  and  2-8.  The  changes  in  the  ac  energy  of  the  Haar 
transform  due  to  movement  of  the  lens  are  localized  in  sequency  groups  11  through 
15,  and  groups  7 through  10.  Also,  both  sequency  groupings  tend  to  behave  in  a 
like  fashion.  It  is  possible,  then,  to  evaluate  the  state  of  focus  of  an  image 
using  only  the  information  contained  in  one  sequency  grouping.  Such  an  approach, 
however,  would  increase  the  focus  measure's  sensitivity  to  image  noise  (particularly 
image  motion)  as  well  as  allow  some  possibly  significant  changes  in  ac  energy  in 
another  group  to  go  unobserved.  For  these  reasons,  the  focus  measures  studied  here 
will  sum  all  of  the  ac  energy  within  the  image's  Haar  transform  and  will  not  single 
out  any  one  sequency  group  or  grouping. 

The  sharpness,  or  sensitivity  to  focus,  of  the  candidate  focus  measures  varies  sub- 
stantially. To  aid  in  comparison,  each  focus  measure  is  plotted  in  Figure  2-12. 

The  curve  resulting  from  the  sum  of  the  magnitudes  of  ac  coefficients  is  comparative- 
ly flat,  its  performance  being  only  slightly  poorer  at  its  end  points  than  that  of 
the  sum  of  the  normalized  magnitudes  of  the  ac  coefficients.  The  relative  insen- 
sitivity exhibited  by  these  two  curves  is  due  to  the  uniform  weighting  of  each 
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(a)  LENS  POSITION  A 


lb)  LENS  POSITION  B 


(c)  LENS  POSITION  C 


Id)  LENS  POSITION  D 


If)  LENS  POSITION  F 


(e)  LENS  POSITION  E 


(g)  LENS  POSITION  G 


hi  LENS  POSITION  H 


(k)  LENS  POSITION  K (I)  LENS  POSITION  L 

Figure  2-5.  TEST  PATTERN  IMAGES  AS  A FUNCTION  OF  LENS  POSITION  (Sli 


(i)  LENS  POSITION  I 


(j>  LENS  POSITION  J 


Figure  2-5.  TEST  PATTERN  IMAGES  AS  A FUNCTION  OF  LENS  POSITION  (Sheet  3 of  3) 


coefficient  within  any  sequency  group.  When  any  set  of  numbers  is  summed  in  this 
fashion,  a substantial  change  in  a few  members  of  the  set  often  is  obscured  by 
effectively  averaging  them  with  the  others.  By  squaring  the  numbers  before  summing 
them,  however,  the  sensitivity  to  small  changes  may  be  increased.  This  increased 
sensitivity  is  exhibited  in  Figure  2-12  by  the  curve  for  the  ac  power  contained  in 
the  image's  Haar  transform. 


Examination  of  the  noise  characteristics  of  each  focus  measure  (Figures  2-9,  2-10, 
and  2-11)  shows  that  their  noise  levels,  expressed  as  a fraction  of  peak  value,  are 
roughly  equivalent.  By  assuming  the  maximum  observed  value  of  a focus  measure  minus 
its  minimum  value  to  be  the  available  signal,  the  signal-to-noise  ratio  can  be 
determined.  The  signal-to-noise  ratios  of  the  first  three  focus  measures  are  shown 
in  Table  2-1.  As  expected,  C,  the  ac  power  of  the  image's  Haar  transform,  has  the 
highest  signal-to-noise  ratio. 


The  behavior  of  texture  and  edge  activity  as  a function  of  lens  position  was  deter- 
mined in  a fashion  similar  to  the  other  focus  measures,  i.e.,  evaluating  the  focus 
measure  at  each  point  through  one  lens  cycle.  However,  the  area  of  the  test  pattern 
for  which  the  descriptors  were  computed  was  different  than  that  used  for  the  other 
measures.  Figure  2-13  shows  the  test  pattern  with  the  descriptor  input  areas  flagged 
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GROUP 

(A)  RMS  NOISE  VERSUS  GROUP 


GROUPINGS 

(B)  RMS  NOISE  VERSUS  GROUPINGS 


Figure  2-9.  RMS  NOISE  LEVELS  FOR  S , SUM  OF  AC  MAGNITUDES 


GROUP 

(A)  RMS  NOISE  VERSUS  GROUP 


GROUPINGS 

(B)  RMS  NOISE  VERSUS  GROUPINGS 


Figure  2-10.  RMS  NOISE  LEVELS  FOR  S , SUM  OF  NORMALIZED  AC  MAGNITUDES 
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Figure  2-12.  A COMPARISON  OF  FOCUS  MEASURES 


Table  2-1.  FOCUS  MEASURE  SIGNAL-TO-NOISE  RATIOS 


Signal- to-Noise 

Focus  Measure 

Ratio  (dB) 

S^,  Sum  of  AC  Magnitudes 

43 

Sn>  Sum  of  Normalized  AC  Magnitudes 

44 

C,  AC  Power 

49 

by  Increased  brightness.  These  areas  were  chosen  so  that  texture  and  edge  would  be 
computed  for  both  the  center  of  the  image,  which  is  primarily  texture,  and  the 
borders  of  the  image,  which  are  composed  primarily  of  two  singular  edges. 

The  computed  values  of  texture  activity,  edge  activity,  and  their  sum  are  shown  as 
a function  of  lens  position  in  Figure  2-14.  As  depicted  in  Figure  2-14,  edge  activ- 
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TEST  PATTERN  DESCRIPTOR  INPUT  AREAS 


TEXTURE  ACTIVITY 
EDGE  ACTIVITY 
ACTIVITY 


LENS  POSITION 


ACTIVITY  DESCRIPTORS  AS  A FUNCTION  OF  LENS  POSITION 


ity  behaves  much  like  ac  power.  Texture  activity,  however,  exhibits  markedly  dif- 
ferent characteristics.  While  similar  to  the  other  curves  in  having  a peak  at  lens 
position  G,  the  texture  activity  curve  also  has  a local  minimum  at  lens  position  C. 
The  precise  effect  that  defocusing  an  image  has  on  the  division  of  activity  into 
texture  and  edge  information  is  not  yet  fully  understood. 

The  computational  complexity  of  each  of  the  focus  measures  is  tabulated  in  Table 
2-2.  The  computation  time  of  each  measure  is  the  time  required  for  the  instrumenta- 
tion system's  minicomputer  to  evaluate  the  state  of  focus  of  one  16  by  16  pel  block 
of  the  image.  The  minicomputer  (Computer  Automation  LSI-2)  has  an  average  instruc- 
tion cycle  time  of  1 psec.  System  overhead  operations  are  not  included  in  the  com- 
putation time  values. 

As  shown  in  Table  2-2,  S is  the  simplest  measure  to  compute  as  it  evolves  only  254 

m 

additions  operations.  S^  requires  190  power  of  two  multiplications  (shifts)  as 
well  as  the  additions  operations.  In  order  to  compute  C,  the  total  ac  power  of  the 
16  by  16  pel  block,  each  of  the  255  ac  coefficients  must  be  squared  before  summation 
occurs.  Because  of  the  LSI-2' s arithmetic  format,  squaring,  or  simple  integer 
multiplication,  uses  two  computer  instruction  cycles.  The  number  of  cycles  re- 
quired to  compute  C then  is  (255  x 2)  + 254  = 765. 

Unlike  the  previous  three  focus  measures,  which  involve  integer  arithmetic,  texture 
and  edge  activity  require  the  use  of  floating  point  arithmetic.  As  a consequence, 
not  only  is  the  number  of  requisite  instruction  cycles  much  greater  for  each  opera- 


Table  2-2.  FOCUS  MEASURE  COMPUTATIONAL  COMPLEXITY 


Measure 

Computation 

Time 

Real  Time 
Compatibility 

S , Sum  of  AC  Magnitudes 
m 

S , Sum  of  Normalized  AC  Magnitudes 
n 

C,  Total  AC  Power 

Texture  and  Edge  Activity 

254  psec 

464  psec 

765  psec 

~204  msec 

Excellent 

Good 

Good 

Very  Poor 
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tlon,  It  la  data  dependent  as  well.  The  computation  time  for  texture  and  edge 
activity  listed  In  Table  2-2  Is  the  time  observed  for  the  computation  of  these  meas- 
ures, scaled  down  by  an  estimate  of  percent  system  overhead.  The  texture  and  edge 
activity  focus  measure  Is  clearly  not  compatible  with  a real-time  system. 

2.3.3  Summary 

The  performance  of  each  focus  measure  In  evaluating  the  state  pf  focus  of  the  video 
test  pattern  Is  summarized  In  Table  2-3. 


Table  2-3.  PERFORMANCE  OF  CANDIDATE  FOCUS  MEASURES 


Measure 

Sensitivity 
to  Focus 

Signal-to-Noise 
Ratio  (dB) 

Required 

Computation 

Time 

S , Sum  of  AC  Magnitudes 
m 

Fair 

43 

254  psec 

S , Sum  of  Normalized  AC 
"Magnitudes 

Fair 

44 

464  psec 

C,  AC  Power 

Good 

49 

756  psec 

texture  vs  Edge 

Poor 

— 

204  msec 

Of  the  measures  studied,  the  total  ac  power  provides  the  focus  measure  with  the 

best  sensitivity  to  focus  and  the  highest  signal-to-noiBe  ratio.  The  only  dis- 

advantage^associated  with  using  ac  power  as  a measure  of  focus  is  that,  compared 

to  S and  S , it  Is  relatively  expensive  to  compute.  In  terms  of  TV  frame  rates, 
m n 

however.  Its  performance  Is  still  acceptable.  For  example,  the  ac  power  of  about 
forty  16  by  16  pel  image  blocks  could  be  computed  during  the  elapsed  time  of  one 
TV  frame. 

The  measures  of  focus  to  be  evaluated  on  the  NVL  FLIR  Imagery  in  the  next  experi- 
ment are  C,  the  total  ac  power  or  image  activity,  and  X and  E,  the  texture  and  edge 
activities  of  the  image.  C was  selected  for  its  superior  performance  on  the  video 
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test  pattern.  Texture  and  edge  activity  are  being  evaluated  further  to  determine 
their  behavior  on  images  that  are  of  a more  general  nature  than  the  video  test 
pattern. 


2.4  FLIR  IMAGE  EXPERIMENT 

In  order  to  ascertain  the  ability  of  ac  power  to  discriminate  focus  on  FLIR  imagery, 
the  five  FLIR  images  supplied  by  NVL  were  used  as  input  data  in  the  test  configura- 
tion shown  in  Figure  2-1.  In  each  image,  areas  containing  only  background  and 
areas  containing  both  background  and  target  information  were  considered.  For  the 
background  data,  the  ac  power  of  three  fixed  sets  of  eight  16  by  16  pel  blocks  was 
evaluated.  These  input  areas  are  flagged  by  increased  brightness  in  Figure  2-15. 

Target  data  were  collected  by  restricting  the  input  data  to  only  the  sector(s)  that 
contained  one  of  the  targets  in  the  image  under  consideration. 

In  each  case,  the  ac  power  of  the  image  input  area  was  evaluated  through  four  lens 
cycles  and  the  results  averaged  for  graphical  presentation. 

j I 

2.4.1  Image  Data 

The  variations  in  ac  power  as  a function  of  lens  position  are  shown  in  Figures  2-16 
through  2-21  for  background  data,  and  in  Figures  2-22  through  2-26  for  target  data. 

As  can  be  seen  in  Figures  2-16  through  2-21,  the  background  data  curves  are  rela- 
tively insensitive  to  focus.  The  curve  for  Image  5 (Figure  2-20),  as  a worst  case 
example,  is  almost  completely  insensitive  to  changes  in  lens  position.  This  effect 
is  due  primarily  to  the  lack  of  activity  in  the  input  sectors  of  the  image. 


Li 


By  choosing  an  alternate  set  of  input  sectors  for  Image  5 and  evaluating  the  result- 
ant fluctuations  in  ac  power,  a curve  whose  sensitivity  to  focus  is  comparable  to 
those  of  the  other  images  is  obtained  (Figure  2-21).  The  alternate  sectors  used 
are  those  immediately  above  the  ones  shown  in  Figure  2-15.  The  average  activity 
increase  obtained  by  moving  the  input  areas  in  this  instance  is  33  percent.  The 
central  input  area  experiences  a two-fold  increase  in  activity. 
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LENS  POSITION 
(B)  AVERAGE  OF  (A) 


Figure  2-18.  TOTAL  AC  POWER  - IMAGE  NO.  3 


The  curves  relating  to  the  state  of  focus  of  the  targets  (Figures  2-22  through 
2-26)  exhibit  an  increased  sensitivity  to  focus  when  compared  to  the  background 
curves.  This  improvement  in  sensitivity  is,  again,  primarily  due  to  the  increased 
activity  of  the  target  sectors.  The  ac  power  of  the  target  sectors  is  about  twice 
that  of  the  average  background  sector.  The  target  curves  showing  the  most  sensi- 
tivity to  focus  are  those  from  Images  1,  2,  and  3.  The  targets  in  each  of  these 
images  are  small,  detailed,  high-contrast  objects.  Objects  of  this  type  appear  to 
provide  the  best  information  on  which  to  base  a state  of  focus  evaluation. 

To  determine  the  effect  of  focus  on  the  division  of  ac  energy  into  texture  and  edge 
activity,  the  image  descriptors  were  evaluated  on  representative  areas  of  selected 
FLIR  images.  The  test  FLIR  imagery  shown  in  Figure  2-2  falls  into  two  major  groups. 
One  group  (Images  1,  2,  and  3)  exhibits  small  targets  on  a relatively  bright  back- 
ground, while  the  other  group  (Images  4 and  5)  consists  of  larger  targets  against  a 
relatively  dark  background.  One  image  from  each  group  (Images  1 and  4)  was  selected 
to  represent  its  group  for  the  evaluation  of  the  image  descriptors.  Both  target  and 
background  areas  of  the  two  images  were  examined.  The  background  areas  are  identical 
to  those  shown  in  Figure  2-15.  For  target  input  information,  a 3 by  4 block  of  image 
sectors  was  centered  about  one  of  the  targets  in  each  image. 

The  curves  relating  the  behavior  of  texture  and  edge  activity  as  a function  of  lens 
position  are  shown  in  Figures  2-27  through  2-30.  It  can  be  seen  that  while  texture 
and  edge  activity  show  approximately  the  same  sensitivity  to  focus,  regardless  of  the 
type  of  input  data,  the  texture  curve  is  much  noisier.  Averaging  the  values  of  tex- 
ture activity  over  several  lens  cycles  would  probably  reduce  this  noise  level  substan- 
tially, but  the  increase  in  computation  time  for  such  an  approach  would  be  prohibitive 
in  a real-time  system.  For  the  background  type  data  seen  in  the  sample  FLIR  images 
provided  by  NVL,  the  division  of  activity  into  texture  and  edge  information  provides 
little  to  no  additional  information  about  the  image's  state  of  focus. 


2.4.2  Summary  of  FLIR  Image  Experiments 

The  ac  power  of  selected  image  sectors  can  provide  a useful  index  of  the  state  of 
focus  of  a FLIR  image.  The  sensitivity  of  the  measure  is  less  for  the  FLIR  images 
than  that  seen  on  the  test  pattern,  but  is  still  sufficient  to  form  the  basis  of 
an  error  signal  in  an  operational  servo  system. 

; 
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Figure  2-23.  TOTAL  AC  POWER  - IMAGE  NO.  2 TARGET 
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Figure  2-28.  ACTIVITY  DESCRIPTORS  AS  A FUNCTION  OF  LENS  POSITION  - 

IMAGE  NO.  1 TARGET 
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Figure  2-29.  ACTIVITY  DESCRIPTORS  AS  A FUNCTION  OF  LENS  POSITION  - 

IMAGE  NO.  4 BACKGROUND 
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Figure  2-30.  ACTIVITY  DESCRIPTORS  AS  A FUNCTION  OF  LENS  POSITION  - 

IMAGE  NO.  4 TARGET 

The  sensitivity  of  such  a measure  to  focus  is  seen  to  be  activity  dependent;  i.e., 
the  ac  power  of  a sector  does  not  perform  well  as  a focus  measure  when  the  maximum 
activity  of  the  sector,  regardless  of  focus,  is  low.  This  result  of  not  focusing 
well  on  low  activity  image  areas  agrees  with  the  manner  in  which  an  operator  would 
focus  an  image.  Rather  than  examining,  say,  the  sky  or  a dim  background  while 
adjusting  the  lens,  he  would  look  at  the  same  sharp  edge  or  fine  texture  to  deter- 
mine focus.  In  both  the  human  and  automatic  cases,  sufficient  ac  high-frequency 
energy  must  be  locally  present  to  evaluate  an  image's  state  of  focus. 

If,  in  the  case  of  automated  focusing,  each  subarea  of  the  image  in  question  cannot 
be  examined  ir.  a reasonable  time,  then  a criterion  of  minimal  activity  would  have  to 
be  established.  Only  those  subareas  satisfying  such  a criterion  would  then  be  con- 
sidered as  useful  input  data  to  the  focus  evaluating  device. 

The  division  of  activity  information  into  texture  and  edge  appears  to  be  of  little 
use  in  evaluating  the  state  of  focus  of  an  image.  The  behavior  of  both  texture 
and  edge  activity  as  well  as  those  of  the  other  image  descriptors  will  continue  to 
be  evaluated  in  subsequent  experiments. 
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The  local  ac  power,  or  activity,  of  an  image  demonstrates  each  of  the  five  charac- 
teristics desired  of  a focus  measure  presented  in  paragraph  2.1.  It  can  be  com- 
puted in  a time  span  that  is  compatible  with  a real  time  system;  its  sensitivity 
to  focus  can  rival  that  of  an  observer:  it  has  a good  signal-to-noise  ratio;  it 
exhibits  scene  Independence,  subject  to  the  limitations  set  forth  a priori;  and, 
since  it  is  a simple  scalar  value  that  may  be  computed  for  each  image  frame,  it  is 
easily  adaptable  to  an  operational  system.  A real  time  imaging  system  that  has  an 
auto  focus  capability  is  certainly  feasible  and  easily  implementable  using  the  two 
dimensional  sensor  processor. 
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SECTION  3 

AUTO  DEBLUR  EXPERIMENTS 


Forward  Looking  Infrared  (FLIR)  images  may  suffer  from  forms  of  blur  not  related  to 
focus.  For  example,  platform  motion  and  the  limited  optical  aperture  may  also  con- 
tribute to  the  blurring  of  the  FLIR  image.  Automated  deblurring  of  FLIR  imagery 
would  therefore  be  a desirable  feature  of  an  advanced  FLIR  system. 

Automatic  image  deblurring  requires  a characterization  of  the  blur  process  and  a 
method  of  inverting  the  blur.  Blur  characterization  is  treated  first.  A method 
of  formulating  an  equivalent  blur  operation  in  an  image's  Haar  domain  is  then 
developed,  and  the  resulting  operations  presented.  Approximations  of  the  blur  oper- 
ations and  deblur  operations  are  computed  and  tested. 

3.1  ONE  DIMENSIONAL  BLUR  MODEL 


The  image  and  blur  models  presented  here  are  one  dimensional.  Many  of  the  results 
presented  are  easily  extended  to  two  dimensions  because  of  separability  considera- 
tions. However,  others  will  require  more  effort.  The  extension  to  two  dimensions 
will  therefore  be  treated  separately. 


Let  x be  an  arbitrary  16  x 1 data  vector.  A shift  invariant  operation  (blur)  is 
to  be  performed  on  x.  The  sampled  values  of  the  impulse  response  of  the  operation 
are  given  by  the  16  x 1 vector  b.  If  £ is  the  output  vector  then 


Z = 2S  * b or  y . 


£ x.b. 

j J ^ 


(3-1) 


where  the  subtraction  of  the  subscripts  is  performed  modulo  16.  Equation  (3-1)  may 
also  be  expressed  in  more  convenient  matrix  format  by 


£ = B x 


(3-2) 


where  the  elements  of  B are  given  by  B. ^ 


b.  . . 
J-i 
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The  operator  matrix  B may  represent  any  shift  invariant  operation  (motion  blur, 
focus  blur,  diffraction  limiting,  laplacian  operator,  etc.).  The  question  we  are 
addressing  is:  How  does  B manifest  itself  in  the  Haar  domain? 


By  taking  the  Haar  transform  of  equation  (3-2)  we  arrive  at 
H^r  = H B x 


or 


Hi  = | H B H_1Jh  x (3-3) 

The  quantity  in  brackets  in  equation  (3-3)  is  a matrix  that  operates  on  the  Haar 
transform  of  the  input  (a  vector)  to  produce  the  Haar  transform  of  the  output  (also  a 
vector).  This  operation  is  independent  of  x and  i and  is  a function  of  only  B.  Let 

M = H B H_1  (3-4) 

denote  this  mapping  of  the  Haar  transform  of  unblurred  original  into  the  Haar  trans- 
form of  the  blurred  image.  Analysis  of  the  forms  of  M obtained  by  the  above  rela- 
tion may  be  facilitated  by  dividing  M into  submatrices  or  groups.  A natural  group- 
ing to  use  with  the  Haar  transform  is  shown  in  Table  3-1.  The  groupings  shown  in 
the  table  are  those  usually  seen  for  the  two-dimensional  Haar  transform.  However, 
the  interpretation  of  "group,"  as  used  here,  is  much  different.  As  a consequence, 
the  group  numbering  scheme  is  also  different. 

The  matrices  that  result  from  equation  (3-4)  are  operators  on  the  one-dimensional 
(1-D)  Haar  domain.  The  groups  of  elements  within  the  operator  will  therefore  be 
referred  to  as  "operator  groups."  The  numbering  system  shown  in  Table  3-1  relates 
the  effect  of  one  of  the  five  sequency  groups  in  the  input  vector  upon  another  se- 
quency  group  in  the  output  vector.  For  example,  if  an  operator  group  is  numbered 
3,5  then  the  elements  of  that  operator  group  represent  the  contribution  of  sequency 
group  3 in  the  input  to- sequency  group  5 in  the  output. 

In  the  experiments  that  follow,  several  forms  of  B and  their  resultant  values  of 
M are  computed.  Also,  some  of  the  effects  of  M on  the  Haar  domain  are  discussed. 
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Table  3-1.  GROUPING  OF  THE  HAAR  DOMAIN  OPERATOR 


_LJ_ 

2,1 

3,1 

4,1 

5,1 

1.2 

2,2 

3,2 

4,2 

5,2 

1,3 

2,3 

3,3 

4,3 

5,3 

1,4 

2,4 

3,4 

4,4 

5,4 

1,5 

2,5 

3,5 

4,5 

5,5 

■ 

3.2  BLUR  OPERATIONS 


The  image  operations  that  were  examined  are: 

a.  Focus  error  (2  cases) 

b.  Diffraction  limited  aperture  (2  cases) 

c.  One-  and  two-pel*  translation. 

Each  is  examined  independently.  In  any  actual  FLIR  system,  however,  they  would 
most  likely  be  encountered  in  various  combinations.  The  translation  operations 
blur  the  display  by  jittering  the  successive  image  frames. 

3.2.1  Focus  Error 

Two  cases  of  focus  error  were  evaluated.  The  characteristic  shape  of  the  impulse 
response  of  a one-dimensional  focus  error  is  a rectangular  pulse  with  unit  area. 


*pel  = picture  element 


The  sampled  values  of  che  two  impulse  responses  used  are  shown  in  Figure  3-1.  The 
Haar  domain  operators  that  result  from  these  impulse  responses  are  shown  in  Figures 
3-2  and  3-3. 


k 
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Examination  of  the  Haar  domain  operators  reveals  many  terms  of  significance  off 
the  main  diagonal  of  the  matrix.  In  fact,  all  of  the  main  diagonal  terms  of  group 
5,5  are  zero  (i.e.,  the  value  of  a high-sequency  term  in  the  output  transform  is 
independent  of  the  value  of  its  corresponding  term  in  the  input  transform).  Also 
note  that  operator  groups  not  residing  on  the  main  diagonal  contain  relatively  large 
terms.  These  results  have  an  impact  on  automatic  image  deblurring  in  several  ways. 
First,  since  the  Haar  domain  operator  M is  poorly  approximated  by  a diagonal  matrix, 
blur  detection  algorithms  may  tend  to  be  more  complex  than  those  in  operating  other 
transform  domains.  Second,  deblurring  approaches  in  the  Haar  domain  will  exhibit 
complexity  comparable  to  that  of  M. 

3.2.2  Diffraction  Limited  Aperture 


Two  impulse  responses  characteristic  of  a diffraction-limited  optical  system  were 
evaluated.  Squared  synchronization  (sync)  functions  were  used  in  order  to  simulate 
the  effects  of  an  incoherently  illuminated  aperture.  Equation  (3-5)  gives  the 
sampled  values  of  both  impulse  responses. 


FOCUS  ERROR  CASE  1 
3 PELS  WIDE 


FOCUS  ERROR  CASE  2 
5 PELS  WIDE 
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Figure  3-1.  IMPULSE  RESPONSES  OF  FOCUS  ERROR 
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3-2.  HAAR  DOMAIN  OPERATOR  FOCUS  ERROR  (CASE  1) 


Case  1 


1 


b.  = 
1 


sync2  ji  it/ 2 J , i = 0,  7 
sync2  | ( 8 - i)  tt / 2 J , i = 8,  15 


(3-5a) 


Case  2 


b.  = 
i 


sync  |0.8  i tt / 2 J , i = 0,  7 
sync2  |(8  - i)  0.8  xr/2  ] , i = 8,  15 


( 3— 5b ) 


Case  1 simulates  a system  sampled  at  the  Nyquist  frequency.  Case  2 represents 
a system  that  is  sampled  25  percent  faster  than  the  Nyquist  frequency.  The  Haar 
domain  operators  that  result  from  these  two  image  operations  are  shown  in  Figures 
3-4  and  3-5. 


Examination  of  Figures  3-3  and  3-4  shows  that  while  the  number  of  nonzero  terms 
in  the  matrices  is  greater  than  in  the  case  of  focus  error,  the  terms  of  the  main 
diagonal  are  generally  greater  than  their  neighbors.  Thus,  the  forms  of  M exhibited 
in  Figures  3-4  and  3-5  appear  to  more  closely  approximate  a diagonal  operator  than 
those  shown  for  focus  error. 


Comparison  of  the  two  operators  shows  Case  1 to  have  a greater  concentration  of 
energy  along  its  diagonal  than  Case  2.  This  is  true  particularly  in  the  highest 
sequences.  The  possibility  of  easily  correcting  for  Case  1 diffraction  limiting 
is  much  greater  than  for  Case  2. 


3.2.3  Translation 


The  Haar  domain  operators  resulting  from  one-  and  two-pel  rigid  translations  are 
shown  in  Figures  3-6  and  3-7,  respectively.  In  each  case,  translation  was  to  the 
right  in  the  image-row  vector  (or  down  in  a column  vector). 
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Figure  3-5.  HAAR  DOMAIN  OPERATOR  APERTURE  BLUR  (CASE  2) 


Figure  3-6.  HAAR  DOMAIN  OPERATOR  ONE-PEL  TRANSLATION 


Figure  3-7.  HAAR  DOMAIN  OPERATOR  TWO-PEL  TRANSLATION 
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Examination  of  Figure  3-6  shows  that  some  of  the  highest  sequency  terms  in  the  out- 
put Haar  spectrum  use  significant  information  from  the  lowest  ac  sequency  in  the 
input  Haar  spectrum.  This  is  shown  immediately  by  the  _+l ' s in  operator  group  2,5. 
This}  form  of  M shows  that  translation  invariant  measures  that  apply  equally  to  all 
of  the  high-sequency  Haar  coefficients  are  at  best  very  complex.  The  possibility 
of  detecting  and  correcting  image  motion  in  the  Haar  domain  is  quite  good,  however. 
This  is  demonstrated  by  the  sparseness  of  the  two  Haar  domain  operators. 

Figure  3-7,  the  Haar  domain  operator  resulting  from  a two-pel  translation,  shows 
an  interesting  result.  The  form  of  operator  group  5,5  is  simply  that  of  a one- 
element  translation  operation.  Thus,  translating  the  image  by  two  pels  translates 
the  highest  sequency  group  of  the  image's  Haar  transform  by  one  term.  Also  note 
that  the  upper  left  quarter  of  Figure  3-7  is  just  a scaled  down  copy  of  Figure  3-6. 
Thus,  translating  two  pels  operates  on  the  middle  sequencies  similar  to  the  manner 
by  which  translating  one  pel  operates  on  the  higher  sequencies. 

3.2.4  Summary 

Each  of  the  Haar  domain  operators  presented  in  this  series  of  experiments  is  non- 
diagonal in  form.  The  net  impact  of  this  characteristic  is  to  make  both  blur  de- 
tection and  inversion  processes  more  difficult.  A diagonal  representation  of  the 
blur  process  is  required.  Such  diagonal  operators  would  lend  themselves  more 
directly  to  simple  detection  and  inversion  methods. 

3.3  A DIAGONAL  REPRESENTATION  OF  BLUR 

A straightforward  method  of  approximating  a Haar  domain  operation,  M,  with  a diag- 
onal matrix  is  to  simply  ignore  those  terms  in  M that  are  off  the  main  diagonal. 
This  approach  has  been  shown  to  not  function  well. 

Another  approach  to  developing  a diagonal  representation  of  a Haar  domain  operator 
is  to  require  that,  on  the  average,  the  effect  of  using  M is  equivalent  to  using 
its  diagonal  representation. 

^D.  J.  Ketcham,  et  al  , Image  Enhancement  for  Cockpit  Displays,  Office  of 
Naval  Research,  0NR-CR2 13-1 24-2 . 
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represent  the  blur  operation.  x ^ is  a Haar  transform  of  an  image  before  blurring, 


M is  the  Haar  domain  operator  representing  the  blur,  and  ^ is  the  resultant  Haar 
transform  of  the  blurred  image.  We  wish  to  find  a positive-valued  diagonal  opera- 
tor, D,  so  that  if 


y.  = D x. 

■*-i  —l 


(3-7) 


the  average  magnitude  of  the  elements  of  will  be  equivalent  to  those  of  ; i.e., 


Eil£ilf  = EIM 


(3-8) 


where  E is  the  expectation  operator  over  all  i and  |^|is  a vector  whose  elements 
are  the  magnitudes  of  ^ . Using  the  requirement  of  equation  (3-8),  D may  then  be 
derived  from  equation  (3-7). 


liil  = o|*il 


(3-9) 


Taking  the  expected  value, 


E{|£ilf  = D E{|x.|| 


(3-10) 


From  equation  (3-8), 


Ej|Z.|f  = D Ej|x.|} 


(3-11) 


Equation  (3-11)  is  saying  that  the  diagonal  operator,  D,  simply  multiplies  the  j 


. th 


element  of  the  average  magnitude  of  the  original  transform  by  a scalor,  d..,  to 

t h ^ ^ 

arrive  at  a value  for  the  j element  of  the  average  magnitude  of  the  blurred  trans- 


form. The  values  of  the  elements  of  D along  its  diagonal,  djj>  are  thus  given  by 
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or  d . . 
JJ 


(3-12) 


E{l*ilf  . 
‘jj  = E||2S-I}J 


E||Mx.|} 


Preliminary  results  of  a series  of  experiments  are  presented  in  which  the  diagonal 
representations  of  the  Haar  domain  operators  presented  earlier  were  computed. 

The  one-dimensional  images  used  as  input  for  these  experiments  were  generated  by 
a first-order  Markov  process  (p  = 0.03).  Two  thousand  such  vectors  were  used  to 
evaluate  the  expected  values  of  Haar  spectra  required  in  equation  (3-12). 

3.3.1  Focus  Error 


The  values  of  d . . observed  for  Case  1 and  Case  2 focus  error  are  shown  in  Figure 
JJ 

3-8.  As  shown  in  the  figure,  the  attenuation  introduced  by  d^  is  greater  for  the 
5-pel-wide  focus  error  than  for  the  3-pel-wide  focus  error.  The  Haar  sequency 


x x X 


X X X X 


CASE  1 FOCUS 
CASE  2 FOCUS 


OOOOOO 


XXXXXX  XX 


000000  oo 


10  11  12  13  14  15  16 


Figure  3-8.  SEQUENCY  ATTENUATION  AS  A RESULT  OF  FOCUS  ERROR 
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spectrum  is  thus  altered  by  a focus  error  in  a manner  similar  to  attenuation  that 
would  be  observed  in  the  frequency  spectrum. 

3.3.2  Limited  Aperture 

The  diagonal  operator  elements  computed  for  Case  1 and  Case  2 aperture  limiting 
are  shown  in  Figure  3-9,  which  shows  that  this  result  of  reducing  the  extent  of 
the  aperture  by  25  percent  is  to  attenuate  the  sequency  spectrum  by  only  a small 
amount.  No  "cutoff  sequency"  analogous  to  a cutoff  frequency  is  observed. 

X - CASE  1 APERTURE 
O - CASE  2 APERTURE 

5 X X 

0 0 X X X X 

0 0 0 0 

xxxxxxxx 

00000000 


„ 1 — I — I I I I I I I I I 1 I I 1 * 

1 2 3 4 5 6 7 8 9 10  11  12  13  14  15  16  j 

4483-25 

Figure  3-9.  SEQUENCY  ATTENUATION  AS  A RESULT  OF  LIMITED  APERTURE 
3.3.3  Image  Motion 

For  both  cases  of  image  motion,  the  elements  along  the  diagonal  of  D are  all  equal 
to  unity.  Although  translating  the  image  introduces  no  sequency  attenuation, 
this  result  is  still  consistent  with  the  blur  model.  Image  motion  is  seen  as 
a blur,  not  within  a specific  frame  of  a display,  but  rather  is  observed  as  a blur 
temporally  from  frame  to  frame.  The  intraframe  effects  of  motion  then  are  not 
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expected  to  alter  the  average  magnitude  of  a single  image's  Haar  transform.  As 
a consequence,  image  motion  will  be  considered  apart  from  the  other  blur  types. 

3.4  APPROXIMATE  DIAGONAL  DEBLUR 

A diagonal  representation,  D,  of  the  blur  operations  has  been  derived  from  the 
average  Haar  spectra  of  both  the  original  and  blurred  image.  A diagonal  representa- 
tion of  a corresponding  deblurring  operator  is  now  required.  The  inverse  of  D is 
the  most  likely  candidate  to  perform  the  requisite  deblurring. 

The  computation  of  D and  D * required  knowledge  of  both  the  original  and  blurred 
image.  However,  in  an  automated  deblurring  system,  the  original  is  not  available. 
Before  some  of  the  real-world  aspects  of  evaluating  D are  considered,  it  is  first 
necessary  to  examine  the  performance  D ^ in  deblurring.  This  will  show  whether 
the  pursuit  of  exact  values  of  D is  worthwhile. 


The  performance  of  D as  a deblurring  operator  will  be  tested  on  the  one  dimensional 
Markov  model  used  before.  Recalling  the  blur  model, 

^ = Mx  Denotes  the  mapping  of  the  Haar  spectrum  of  the  original  image 

into  the  Haar  spectrum  of  the  blurred  image, 


£ = Dx 


is  the  diagonal  representation  of  the  blur,  and 


x = D ^ is  the  diagonal  representation  of  the  deblurring  operation. 

Since  we  are  dealing  with  one  dimensional  data,  the  Mean  Squared  Error  (MSE)  between 
two  Haar  vectors  is  used  to  characterize  the  effects  of  the  various  blurring  and 
deblurring  operations. 


a = (x  - y )2 

16  * lXj  V 

be  the  MSE  due  to  blurring  and 


be  the  residual  MSE  observed  after  deblurring. 

-1 

The  results  of  deblurring  with  D are  shown  in  Table  3-2. 

-1 

Table  3-2.  MSE  OBSERVED  USING  D TO  DEBLUR 


Blur  Type 

— 

MSE  Due  to  Blurring 

MSE  After  Deblurring 

Focus  (Case  1) 

0.35 

0.38 

Focus  (Case  2) 

0.62 

0.72 

Aperture  (Case  1) 

0.20 

0.15 

Aperture  (Case  2) 

0.29 

0.24 

As  shown  in  the  table,  the  residual  MSE  is  comparable  to  that  due  to  blurring. 

A slight  improvement  is  seen  in  the  aperture  blur  examples  while  the  focus  blurs 
show  a small  degradation.  The  relative  performance  of  D ' in  undoing  these  two 
types  of  blurs  is  consistent  with  the  observed  characteristics  of  the  exact  Haar 
domain  implementations  computed  previously.  For  example,  from  the  comparison  of 
Figures  3-3  and  3-4,  one  would  expect  the  operator  representing  the  aperture  blur 
to  be  more  closely  approximated  by  a diagonal  operator  than  the  operator  repre- 
senting focus  blur.  The  results  shown  in  Table  3-2  bear  this  out. 

Although  little  to  no  improvement  in  an  MSE  sense  is  realized  by  using  D * to  de- 
blur one  dimensional  images,  a subjective  enhancement  should  be  obtained.  This 
prediction  is  based  on  edge  enhancement  experiments  (Section  4)  showing  that  a 
marked  crispening  effect  is  observed  when  the  high  sequencies  of  an  image's  Haar 
spectrum  are  amplified.  The  use  of  D * to  deblur  will  therefore  be  investigated 
using  some  two  dimensional  images.  The  resultant  improvements,  if  any,  may  then 
be  evaluated  with  the  MSE  as  well  as  subjectively  and  using  image  descriptors. 
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The  problem  of  evaluating  D and  its  inverse  still  remains.  From  equation  (3-12)  we 
see  that  the  average  Haar  spectrum  of  the  original  must  be  known  in  order  to  assess 
the  average  sequency  attenuation  resulting  from  the  blur.  Since  we  are  given  only 
the  blurred  image,  a prototype  original  must  be  assumed.  By  then  calculating  the 
average  magnitude  of  the  Haar  spectrum  of  blurred  image  and  comparing  it  to  that 
of  the  prototype  an  estimate  of  D may  be  made.  This  method  of  substituting  a 
prototype  spectrum  for  that  of  the  original  is  analogous  to  power  spectrum  equali- 
zation method  of  blind  deconvolution.  This  approach  has  been  used  with  good  re- 

2 

suits  by  Stockham  et  al 

The  definition  of  the  prototype  Haar  spectrum  will  establish  the  amount  of  sequency 
attenuation  observed  in  any  one  image.  In  the  one-dimensional  examples  presented 
thus  far,  the  average  magnitude  of  the  input  Haar  spectrum  follows  directly  from  the 
Markov  model  utilized  to  generate  the  image  vectors.  When  dealing  with  real  images, 
however,  the  uniformity  and  stability  of  the  average  magnitude  of  the  Haar  spectrum 
must  still  be  investigated  before  a prototype  substitution  may  be  safely  made. 

For  the  spectrum  prototype  to  be  meaningful,  the  average  magnitude  of  the  Haar  co- 
efficients must  not  vary  substantially  from  image  to  image.  Figure  3-10  shows  six 
FLIR  images  displaying  distinctly  different  scenes.  The  corresponding  average  mag- 
nitude Haar  spectra  for  scan  lines  of  the  image  are  shown  in  Figure  3-11.  Each  of 
the  Haar  spectra  shown  exhibit  the  same  characteristic  shape;  i.e.,  relatively  high 
amplitude,  low  sequency  terms  with  a gradual  tapering  off  with  increasing  sequency. 
By  forming  the  average  of  each  of  the  curves  in  Figure  3-11  and  averaging  across 
sequency  groupings,  the  Haar  spectrum  prototype  can  be  formed.  The  prototype  is 
shown  in  Figure  3-12.  Comparison  of  the  individual  image  curves  with  the  prototype 
shows  that  the  images  vary  by  about  +20  percent  from  the  prototype.  In  light  of  the 
marginal  change  in  MSE  observed  by  deblurring  with  D \ 20  percent  tolerance  in 
prototype  definition  should  be  acceptable. 

3.5  MOTION  DETECTION/CORRECTION 

Motion  blur,  as  considered  here,  is  a uniform  translation  or  "jump"  of  the  image 
from  one  image  frame  to’ the  next.  While  this  effect  does  not  blur  any  individual 
image  frame,  displaying  translated  frames  at  30  hertz  produces  a subjective  smearing 
or  jitter  effect. 

2T.  G.  Stockham,  T.  M.  Cannon,  R.  B.  Ingebretsen  "Blind  Deconvolution  Through 
Digital  Signal  Processing,"  Proc.  IEEE,  4-75,  V63,  No. 4. 


3-18 


(ft  IMAGE  10 


le)  IMAGE  8 


Figure  3-10.  FLIR  THERMOGRAMS 
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(b)  IMAGE  5 


(c)  IMAGE  6 
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Figure  3-12.  HAAR  SPECTRUM  PROTOTYPE 


Since  no  one  frame  is  blurred,  image  motion  must  be  detected  by  comparing  the  Haar 
spectra  of  successive  image  frames.  In  one  dimension,  two  images  are  related  by 
the  Haar  domain  operator  for  motion.  The  two  forms  of  this  operator  resulting  from 
left  and  right  image  translations  are  shown  in  Figures  3-13  and  3-14.  Examination 
of  Figures  3-13  and  3-14  reveals  the  following  characteristics: 

a.  Four  of  the  bottom  eight  rows  of  each  operator  have  only  three  nonzero 
elements  (i.e.,  they  exhibit  a low  complexity). 

b.  The  values  of  these  nonzero  elements  are  1 and  -0.5  and  can  thus  be  easily 
implemented  in  digital  hardware. 


c.  The  rows  w:th  these  properties  are  the  odd-numbered  rows  for  a left  shift 
and  even  numbered  rows  for  a right  shift.  The  effects  of  shifting  either  left  or 
right  are  therefore  separable. 


3-21 


DOMAIN  OPERATOR  FOR  A ONE-PEL  RIGHT  SHIFT 


d.  Each  even-numbered  row  for  the  right  shift  is  identical  to  a neighbor  odd- 
numbered  row  for  the  left  shift  (e.g.,  Row  9 in  Figure  3-14  is  identical  to  Row  10 
in  Figure  3-13).  This  results  in  a beneficial  redundancy. 

These  properties  lead  to  the  motion  detection  method  shown  in  Figure  3-15. 


IMAGE 


Figure  3-15.  MOTION  DETECTION  METHOD 
The  box  "Compute  Simple  Rows"  performs  the  following  function: 


T[m/2]  ' ar-l,  [m/21 


- I (a  + a A 
2 y r ,m  r ,m-l  f 


(3-13) 


where  r and  m are  the  sequency  and  offset  of  the  Haar  function  and  [m/2]  means  "the 
integer  part  of  m/2." 
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The  comparators  then  compare  values  of  T[m/2)  w^t^1  t^le  corresponding  odd  and  even 

a . If  they  are  equal  (within  a certain  tolerance),  a shift  in  the  appropriate 
r ,m  ^ 

direction  is  indicated. 

This  method  of  motion  detection  involves  16  adds  and  8 comparisons  to  detect  a left 
or  right  shift.  An  equivalent  method  in  the  image  domain  would  require  32  com- 
parisons. A potential  computational  savings  could  thus  be  realized  by  using  this 
approach . 

Once  an  image  shift  has  been  detected,  the  motion  may  be  easily  reversed  by  an 
appropriate  inverse  shift  in  the  image  domain. 

3.6  TWO  DIMENSIONAL  BLUR  MODEL 

The  results  presented  so  far  have  been  one-dimensional.  Real  life  imagery  is,  of 
course,  two-dimensional,  thus  requiring  a two-dimensional  formulation  of  the  blur- 
ring and  deblurring  process.  Let  X be  a 16  x 16  sampled  original  and  Y its  blurred 
image.  The  mapping  of  X into  Y can  be  expressed  as  a two-dimensional  convolution 


{ Y f ij  k 1 bk,l  ^k-i,  1-j 


(3-14) 


where  b^  is  the  impulse  response  of  the  blur.  If  the  blur  is  separable  (i.e.  can 
be  expressed  as  two  one-dimensional  blurs)  then  a more  convenient  matrix  expression 
may  be  used,  namely 


Y = B X B 


(3-15) 


where  B is  the  matrix  representation  of  a one-dimensional  blur.  We  will  assume  for 
the  moment  that  the  blur  is  indeed  separable. 

By  taking  the  two  dimensional  Haar  transform  of  (3-15)  we  get 

T T T 

HYH  = H B X B H 


or,  since  H ^ 
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H Y HT  = |h  B HTj  |h  X HTj  |h  BT  HT 
= |h  B HTj  |h  X HT|  |h  B HTj 


(3-16) 


Recalling  equation  (3-4),  equation  (3-16)  may  be  written  as 


H Y HT  = M | H X HT  j MT 


(3-17) 


where  M is  the  Haar  domain  operator  for  the  blur  in  question.  The  forms  of  M re- 
sulting from  several  typical  blurs  were  presented  in  paragraph  3.2.  Each  operator 
is  highly  non-diagonal  in  nature  and  therefore  complicates  the  blur  detection  and 
inversion  process.  However,  the  diagonal  representation  of  M may  be  substituted, 
thereby  simplifying  the  expression. 


hyht  = d|hxhtJd 


(3-18) 


To  implement  equation  (3-18),  a new  matrix  D can  be  formed  where 


Mu  ■ Mu  H 


(3-19) 


and  the  estimate  of  the  blurred  image  is  given  by 


* T * . f t1 
H Y H = D H X H 


(3-20) 


where  implies  element-by-element  multiplication,  the  matrix  D relates  the  aver- 

age sequency  attenuation  observed  due  to  the  blur.  In  cases  where  the  blur  function 

* 

is  not  separable  into  two  one-dimensional  blurs,  the  elements  of  D may  be  derived 
directly  from  the  two-dimensional  Haar  spectra  of  the  original  and  blurred  images. 


{«  ' "1u 

{^}u 


(3-21) 


where  H Y H denotes  the  average  of  the  magnitude  of  the  blurred  images  Haar  spec- 
if 

trum.  The  assumption  of  separability  is  therefore  not  required  to  evaluate  D . 
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The  reciprocal  of  the  elements  of  D can  now  be  used  to  approximately  deblur  two- 
dimensional  images  in  the  same  manner  that  D ^ was  used  on  one-dimensional  images. 

The  average  magnitude  of  the  input  Haar  spectrum  is  assumed  to  be  known  a priori. 

The  Haar  spectrum  prototypes  computed  in  the  same  fashion  as  in  paragraph  3.5  may  also 
be  substituted  for  the  input  spectrum. 

This  deblurring  process  is  given  by 


T 1 * ^ f T 

HXH  = D " I H Y H 


(3-22) 


where 


‘J  <D\. 


3.7  TEST  IMAGES 

A 256  x 256  pel  region  of  each  of  the  six  1’LIR  images  presented  in  Figure  3-10  was 
blurred  by  the  blur  functions  presented  in  paragraph  3.2.  The  resultant  blurred 
images  were  then  subjected  to  the  deblurring  process  given  by  equation  (3-22).  The 
original,  blurred,  and  deblurred  images  are  presented  in  Figures  3-16  through  3-21 
for  the  case  2 focus  blur.  This  blur  was  chosen  for  pictorial  demonstration  since 
it  is  the  most  severe  in  an  MSE  sense.  A summary  of  the  changes  in  MSE  and  image 
descriptors  for  all  the  blurs  for  each  image  is  presented  in  Table  3-3. 

As  shown  in  Table  3-3,  deblurring  an  image  using  this  approach  results  in  an  MSE 
dig  radar i ,n  of  at  least  three  times  that  observed  from  the  blur  itself.  This  changes 
im-.g.  descriptors,  however,  indicate  a substantial  increase  in  edge  activity  as 
Ich I urring.  These  edges  are  evidenced  in  the  deblurred  images  (Figures 
, . . .i,,t  . r ch.,  detail,  but  rather  as  ringing  and  noise  across  the  entire 


t..i  i i result  of  deblurring  was  not 
a.  i tin  deblurring  process. 
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A.  IMAGE  NO.  4 ORIGINAL 
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A.  IMAGE  NO.  5 ORIGINAL 


B.  BLURRED  FOCUS.  CASE  2 


B.  BLURRED  FOCUS.  CASE  2 
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C.  DEBLURRED 
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DEBLUR  RESULTS  FOR 
IMAGE  It 5. 


DEBLUR  RESULTS  FOR 
IMAGE  //I . 
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A.  IMAGE  NO.  7 ORIGINAL 
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B.  BLURRED  FOCUS.  CASE  2 
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Figure  3-18.  DEBLUR  RESULTS  FOR 
IMAGE  #6. 


Figure  3-19.  DF.BLUR  RESULTS  FOR 
IMAGE  #7. 
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B.  BLURRED  FOCUS.  CASE  2 4697-57 


B.  BLURRED  FOCUS.  CASE  2 4697-60 


C.  DEBLURRED 
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Figure  3-20.  DEBLUR  RESULTS  FOR 
IMAGE  // 8 


Figure  3-21.  DEBLUR  RESULTS  FOR 
IMAGE  HO. 


Table  3-3.  SUMMARY  OF  DEBLUR  RESULTS* 
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MSE  (%)  Edge  Activity 


Texture  Activity,  x 


Texture  Activity 


xQ 


16 

10 

31 

16 


Texture  Activity,  x 


Texture  Activity 


Edge  Activity,  e = 13 


MSE  (%)  Edge  Activity 


Texture  Activity,  x 


Texture  Activity 


XP 


43 


Edge  Activity,  3 


MSE  (%) 


Edge  Activity 


Texture  Activity,  x 


Texture  Activity 


*Subscript  a refers  to  blurred  image,  p to  "deblurred"  image 
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Table  3-3.  SUMMARY  OF  DEBLUR  RESULTS*  (Continued) 


Image  #8  Edge  Activity,  e = 143  Texture  Activity,  x = 51 


MSE 

(%) 

Edge  Activity 

Texture  Activity 

Blur 

a 

P 

ea 

eP 

xo 

XP 

Focus  1 

1.4 

6.5 

in 

236 

19 

42 

Focus  2 

4.3 

13.5 

93 

363 

13 

82 

Aperture  1 

0.5 

5.1 

126 

204 

25 

46 

Aperture  2 

1.3 

5.9 

110 

242 

19 

45 

Image  it  10 

Edge  Activity,  e = 109 

Texture  Activity,  x = 25 

MSE 

(%) 

Edge  Activity 

Texture  Activity 

Blur 

a 

P 

eP 

XP 

Focus  1 

1.2 

6.2 

74 

180 

13 

31 

Focus  2 

3.5 

12.1 

62 

271 

7 

49 

Aperture  1 

0.4 

5.2 

88 

151 

18 

33 

Aperture  2 

1.0 

5.9 

76 

176 

12 

33 

*Subscript  a refers  to  blurred  image,  p to  "deblurred"  image 


When  deblurring,  all  sequency  terms  regardless  of  amplitude  are  amplified  linearly 
by  the  appropriate  element  of  the  deblur  matrix.  The  results  of  the  edge  enhance- 
ment experiments  show  that  subjective  enhancement  is  best  obtained  by  a nonlinear 
increase  of  only  some  above-threshold  sequency  terms. 

3.8  CONCLUSION 


The  results  of  deblurring  with  a diagonal  operator  in  the  Haar  domain  show  a degra- 
dation in  image  quality  (both  subjective  and  MSE)  instead  of  the  desired  improvement. 
This  result  is  due  primarily  to  the  basic  non-diagonal  nature  of  blurs  in  the  Haar 
domain  and  the  rough  approximation  that  a diagonal  blur  operator  provides.  The  Haar 
domain  is,  in  general,  not  well  suited  for  the  implementation  of  scalar  filtering 
functions  that  correspond  to  spatial  convolution. 


The  Haar  transform  can,  however,  provide  a measure  of  the  image  loss  due  to  blur- 
ring. By  comparing  the  observed  average  magnitude  spectrum  with  the  Haar  transforme 
prototype,  some  information  about  the  magnitude,  if  not  the  nature,  of  the  blur  may 
be  obtained.  In  an  automated  deblurring  process,  this  information  would  be  required 
before  any  further  processing  was  initiated. 


SECTION  4 

EDGE  ENHANCEMENT  EXPERIMENTS 


4 . 1 INTRODUCTION 

A fundamental  task  of  imaging  sensor  systems,  whether  thermal  or  visual,  is  to 
facilitate  target  detection  and  classification.  The  target,  being  a man-made  object, 
usually  consists  of  a greater  number  of  straight  lines  (edges)  than  most  natural 
objects.  Therefore,  one  of  the  more  promising  techniques  for  extracting  the  target 
from  natural  background  clutter  is  edge  enhancement.  Edge  enhancement  will  also 
highlight  roads,  overhead  wires,  paths,  and  other  such  cues  as  to  where  a target 
may  be  found.  The  object  of  the  experiment  described  in  this  section  is  to  deter- 
mine how  the  edge-correlating  coefficients  of  the  rationalized  Haar  transform  can 
be  effectively  modified  in  real  time  so  that  these  edges  are  highlighted  and  image 
details  are  made  more  visible. 


During  this  period,  a set  of  edge-enhancement  algorithms  were  developed  and  evalu- 
ated both  subjectively  and  in  terms  of  image  descriptors. 

Edge  enhancement  algorithms  should  exhibit  the  following  characteristics: 


a.  Low  Complexity  - The  computational  complexity  of  the  enhancement  algo- 
rithms must  be  amenable  with  real-time  processing  of  video  imagery. 


b.  Scene  Independence  - The  selected  edge-enhancement  scheme  must  perform 
equally  well  for  different  types  of  scenes.  Edge-enhancement  algorithms  should 
tolerate  both  high-  and  low-contrast  situations  Also,  performance  should  not 

be  significantly  altered  by  image  content;  e.g.,  target  size  and  background  clutter. 

c.  Noise  Immunity  - System  noise  can  be  categorized  into  two  classes: 

(1)  low-frequency  noise,  where  the  grey  shade  levels  of  an  image  vary  from  one  TV 
frame  to  another,  and  (2)  random,  high-frequency  noise  (transient  or  shot  noise). 
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Provisions  should  be  made  in  the  edge-enhancement  algorithm  to  minimize,  if  not 
eliminate,  enhancement  of  these  types  of  system  noise. 

d.  Adaptability  in  the  Operational  System  - The  selected  edge-enhancement 
scheme  must  be  capable  of  being  hardware  implemented  with  sufficient  resolution, 
dynamic  range,  stability,  and  speed  to  be  compatible  with  the  FLIR  image  quality 
parameters  as  specified  in  the  classified  appendix  to  the  First  Quarterly  Report 
(76Y133) . 

4.2  GENERAL  EXPERIMENTAL  PROCEDURE 
The  experiment  was  divided  into  three  major  tasks: 

a.  Preliminary  studies  and  experiments  were  conducted  to  establish  the  basic 
criteria  for  edge  enhancement.  The  goal  of  this  effort  was  to  determine  which 

of  the  15  Haar  sequency  groups  require  alteration  and  the  amount  of  alteration 
necessary  in  order  to  enhance  edges. 

b.  The  performance  of  several  different  methods  of  edge  enhancement  on  a 
representative  FLIR  image  were  examined. 

c.  The  performance  of  the  edge  enhancement  methods  selected  in  the  second 
task  were  evaluated  on  a set  of  six  FLIR  thermograms  supplied  by  NVL. 

The  experimental  test  configuration  is  shown  in  Figure  4-1.  To  ensure  the  validity 
of  the  experimental  test  results,  a number  of  test  setup  procedures  were  initiated 
as  follows: 

a.  To  ensure  that  the  test  image  was  in  focus,  the  lens  of  the  video  camera 
was  manually  adjusted  using  the  autofocus  concept  described  in  Section  2. 


b.  The  f-stop  of  the  video  camera  lens  was  opened  to  its  extreme  (f/1.4). 
This  minimizes  degradation  of  the  system  Modulation  Transfer  Function  (MTF)  by  the 
optics.  Also,  to  prohibit  the  video  signal  from  being  overdriven  into  saturation, 
an  appropriate  neutral-density  filter  was  inserted  between  the  test  image  and  the 
! camera  lens. 


0 

0 

I 
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Figure  4-1.  EXPERIMENTAL  TEST  CONFIGURATION 


c.  The  offset  bias  and  the  video  gain  were  adjusted  to  obtain  the  maximum 
attainable  dynamic  range  for  each  FLIR  test  image. 

The  Image  Descriptors  (ID' s)  of  each  original  and  enhanced  image  used  in  this  study 
were  computed  and  recorded  as  a measure  of  enhancement.  The  ID's  corresponding 
to  each  image  of  a series  of  experiments  are  presented  at  the  end  of  the  appropriate 
presentation. 

The  six  FLIR  thermograms  used  for  the  enhancement  experiments  supplied  by  Night 
Vision  Laboratory  are  shown  in  Figure  4-2.  Their  image  descriptors  are  presented 
in  Table  4-1.  This  subset  of  the  available  data  base  was  selected  to  demonstrate 
performance  of  edge-enhancement  features  over  a range  of  scenes . 

4.3  EDGE  ENHANCEMENT  CRITERIA 

The  edge  terms  of  an  image's  Haar  transform  are  those  that  satisfy  the  edge  criterion 
function  as  defined  in  the  section  on  image  descriptors  in  the  First  Quarterly 
Report.  The  edge  criterion  function  adaptively  sets  an  edge  threshold,  0,  based  on 
the  power  distribution  within  each  sequency  group.  AC  terms,  the  amplitudes  of  which 
are  greater  than  the  threshold,  are  identified  as  edges.  Computation  of  0,  however, 
requires  a large  number  of  floating-point  operations.  The  present  hardware  config- 
urtion  of  the  two-dimensional  sensor  processor  coupled  with  the  requirement  that 
edge-enhancement  algorithms  operate  in  real  time  preclude  the  use  of  such  an  adap- 
tive threshold  for  edge  definition.  In  performing  the  FLIR  image  experiments 
described  in  paragraph  2.5,  the  adaptive  thresholds  were  used  in  non-real  time. 

4.3.1  Definition  of  Edges 

Edge  terms  identified  by  0 can  appear  in  most  of  the  sequency  groups  of  an  image's 
Haar  transform.  However,  only  some  of  sequency  groups  are  appropriate  for  edge 
enhancement . 

The  Haar  coefficient  groups  selected  for  edge  enhancement  must  meet  two  basic  cri- 
teria. First,  the  area  on  which  the  Haar  function  has  an  effect  should  not  be 
any  larger  than  the  smallest  object  of  interest  in  the  scene.  If  this  condition 
is  not  met,  the  object  of  interest  will  simply  take  on  the  shape  of  the  Haar  function 
when  enhanced.  The  second  criterion  is  that  the  area  of  the  Haar  functions  selected 
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(a  IMAGE  1 


(b  IMAGE  5 


(c)  IMAGE  6 


(d)  IMAGE  7 


le)  IMAGE  8 


(f)  IMAGE  10 
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should  be  large  enough  that  random  high-frequency  noise  in  the  system  will  not  be 
enhanced,  creating  random  speckles  on  the  display.  The  two  criteria  thus  place 
upper  and  lower  sequency  bounds  on  the  range  of  Haar  functions  suited  to  edge 
enhancement . 


Figure  4-3  illustrates  the  size  and  shape  of  each  Haar  function  in  group  4 (located 
at  upper  left  corner)  through  group  15  (lower  right  corner).  Subjective  evaluation 
of  the  size  and  shape  of  the  Haar  functions  with  that  of  the  scene  in  the  background 
led  to  the  selection  of  groups  9 through  15,  which  can  be  altered  for  edge  enhancement, 


Separation  of  the  edge  terms  from  non-edge  terms  in  groups  9 through  15  necessitates 
the  derivation  of  an  edge  threshold.  The  sensor  processor  has  the  capability  of 
setting  a different  fixed-edge  threshold  for  each  sequency  group,  j,  of  an  image's 
Haar  transform.  An  immediate  goal  in  design  of  an  edge-enhancement  algorithm  is 
to  determine  a set  of  fixed-edge  thresholds  0^  that  can  approximate  the  adaptive  one. 

As  a first  step  in  obtaining  a fixed  threshold,  the  values  of  the  adaptive  edge 
threshold  derived  by  the  image  descriptor  for  each  individual  sector  were  averaged 
across  each  of  the  six  test  images.  The  resultant  average  thresholds  for  groups  9 
through  15  are  shown  in  Table  2-2.  As  shown  in  the  table,  the  average  value  of 
the  adaptive  edge  threshold,  0,  varies  substantially  from  group  to  group,  but  is 
relatively  constant  from  image  to  image.  This  consistency  in  the  behavior  of  0 
shows  a beneficial  scene  independence  in  the  definition  of  an  edge.  The  values 
of  0 for  some  groups,  however,  are  quite  high.  This  is  because  of  the  dual  function 
that  0 performs.  When  0 is  low,  it  serves  as  an  edge-texture  separator.  When 
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Figure  4-3.  HAAR  FUNCTIONS,  GROUPS  4 THROUGH  15 


Table  4-2.  AVERAGE  ADAPTIVE  THRESHOLD  (IMAGE  1) 


Sequency 

Group 

Average  Adaptive  Threshold  8 

0 Average 

Image 

1 

Image 

5 

Image 

6 

Image 

7 

Image 

8 

Image 

10 

9 

4.2 

3.9 

2.8 

4.2 

1.9 

1.8 

3.1 

10 

1.9 

1.4 

1.2 

1.9 

0.9 

0.9 

1.4 

11 

18.6 

26.7 

14.1 

23.3 

17.6 

11.4 

18.6 

12 

3.1 

3.4 

1.9 

3.3 

1.6 

1.4 

2.4 

13 

4.6 

4.1 

3.4 

5.1 

3.6 

2.7 

• 3.9 

14 

9.3 

7.0 

6.5 

10.5 

6.3 

6.2 

7.6 

15 

10.1 

7.8 

7.4 

12.4 

7.6 

7.5 

8.8 

0 takes  on  large  values,  it  is  preventing  high-power  texture  terms  from  being  con- 
sidered as  edges.  To  fulfill  this  second  function  0 frequently  is  quite  large 
as  reflected  in  the  large  values  of  0.  Since  0 can  exhibit  values  greater  than 
the  maximum  observed  coefficient  amplitudes,  it  is  therefore  difficult  to  derive 
a fixed-edge  threshold  0^  from  the  values  of  0 shown  in  Table  4-2. 

Another  approach  to  determining  valid  fixed-edge  thresholds  is  to  require  that, 
on  the  average,  0.  identifies  an  equal  number  of  edge  terms  as  8.  The  percentages 
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of  edge  terms  observed  in  sequency  groups  9 through  15  in  each  of  the  test  images 
are  given  in  Table  4-3. 

Table  4-3.  PERCENT  EDGE  TERMS 


Image 

Percent  Edge  Terms  Groups  9-15 

1 

13 

5 

23 

6 

22 

7 

12 

8 

18 

10 

21 

The  average  number  of  edge  terms  in  the  higher  sequencies  identified  by  0^  is  about 
18  percent.  Values  of  0^  with  comparable  performance  can  be  determined  from  the 
coefficient  amplitude  histogram  shown  in  Figures  4-4  through  4-8.  A summary  of  the 
percentages  of  terms  exceeding  some  intermediate  values  of  8j  that  approximate  0 
are  shown  in  Table  4-4. 


Table  4-4.  NUMBER  OF  TERMS  EXCEEDING  0.  (IMAGE  1) 

J 


Group  No . 

(j) 

Coefficient  with  Amplitude 

< (0.) 

J 

Probability  of  Exceeding 

0.  (Percent) 

J 

9 

2 

19.4 

10 

2 

19.6 

11 

2 

23.0 

12 

2 

10.5 

13 

2 

16.1 

14 

2 

28.7 

15 

3 

20. 1 

These  values  of  6 will  be  used  as  a starting  point  in  subsequent  experiments.  Al- 
though these  values  of  0.  are  based  on  only  one  image,  examination  of  the  total 
ac  energy  histograms  for  all  six  FLIR  test  images  (Figures  4-9  and  4-10)  shows 
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Figure  4-8.  CROUP  15,  IMAGE  1 

amplitude  distributions  to  be  roughly  identical.  This  leads  us  to  believe  that 
the  enhanceswnt  algorithm  selected  for  one  image  will  perform  equally  well  for 
other  types  of  scenes. 

4.3.2  Weighting  Factors 
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Li 


0 

11 


After  the  edge  infonsation  is  separated  from  texture  and  system  noise,  the  Haar 
coefficients  must  then  be  altered  to  obtain  the  desired  effect  on  the  display. 

To  accomplish  this  goal,  the  weighting  factors  for  the  edge  terms  in  each  group 
of  Haar  coefficients  must  be  determined.  In  addition,  an  upper  limit  of  edge  ampli- 
tude must  be  set  to  avoid  further  enhancement  and  possible  distortion  of  already 
highly  visible  edges. 

Because  the  diswnsions  of  the  Haar  functions  to  be  altered  (Groups  9 through  15) 
vary,  the  apparent  brightness  of  one  group  of  functions  BMy  differ  from  that  of 
another  group  although  their  coefficient  amplitudes  are  the  same.  This  is  illus- 
trated in  Figure  4-11.  The  amplitudes  of  each  of  the  Haar  coefficients  are  equally 
set  to  an  arbitrary  constant.  As  shown  in  the  figure,  the  apparent  brightness 
of  the  corresponding  Haar  functions  degrades  progressively  as  sequency  increases. 
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Figure  4-11.  GROUPS  9 THROUGH  15  WITH  IDENTICAL  AMPLITUDE 

To  ensure  thet  the  edge  information  in  different  Hear  coefficient  groups  hes  equal 
weight  in  edge  enhancement,  e test  was  conducted  to  determine  the  ss^litudes  at 
which  the  Near  functions  from  different  groups  produced  an  equivlent  visibility 
on  the  display.  The  results  of  this  test  are  shown  in  Figure  4-12;  e different 
equivalent  brightness  is  shown  in  each  photograph  of  Figure  4-12.  A tabulation 
of  the  equivalent  brightness  amplitudes  used  in  each  coefficient  group  is  also 
given. 

Edge  coefficients  with  amplitudes  just  above  the  edge  threshold  must  be  altered 
sufficiently  to  crispen  the  image  without  creating  distortions  detrimental  to  the 
eosawtic  appearance  of  the  display.  Minimum  amplitude  edges  should  be  assigned 
values  that  make  them  barely  visible  in  the  display.  For  subsequently  larger  ampli- 
tudes of  edge  terms,  the  assigned  amplitudes  are  increased  until  a point  is  reached 
Where  the  contrast  of  the  target,  or  any  other  object,  is  such  that  enhancement 
ia  no  longer  required.  The  selected  groups  of  barely  visible  Haar  functions  with 
their  corresponding  amplitudes  are  shown  in  Figure  4-12a.  The  amplitudes  of  the 
functions  shown  in  Figure  4-12b  have  been  tentatively  selected  as  the  maximum  aapli- 
tudes  where  enhancesmnt  is  no  longer  preferred.  Figure  4-12c  shows  the  effect 
on  the  iaMgery  when  much  larger  amplitudes  are  used. 

TWo  different  methods  for  altering  the  Haar  coefficients  in  order  to  enhance  edges 
were  selected  for  study.  The  two  methods  are  the  constant  multiplier  and  the  vari- 
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The  constant  multiplier  approach  to  edge  enhancement  employs  only  shift  operations 
to  alter  the  amplitude  values  of  the  selected  Haar  coefficients  deemed  to  be  edge 


terms.  Each  group  of  coefficients  may  be  programmed  to  have  a different  multiply- 
ing factor.  Once  the  multiplying  factor  is  selected  for  that  particular  group, 
all  edge  terms  in  that  group  will  be  altered  by  the  same  multiplying  factor.  This 
method  is  the  simpler  of  the  two  approaches  from  the  standpoint  of  hardware  design 
and  implementation. 
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The  variable  Multiplier  Method  adapts  the  multiplier  based  on  the  amplitude  of 
the  incoming  edge  information;  i.e.,  if  in  one  group,  the  amplitude  of  the  edge 
term  is  3,  the  sniltiplying  factor  would  be  boom  value  different  than  the  multi- 
plying factor  for  an  incoming  edge  term  with  an  amplitude  of  4.  Although  this 
method  of  edge  enhancement  is  more  complex  than  the  previous  approach,  it  is  limited 
only  by  the  amplitude  resolution  of  the  system.  The  variable  multiplier  approach 
effectively  creates  a nonlinear  transfer  function. 

4.4  EDGE  ENHANCEMENT  EXPERIMENT 


In  order  to  evaluate  the  capability  of  the  edge-detection  concepts  and  the  perform- 
ance of  the  two  methods  for  altering  edge  terms,  experiments  were  conducted  using 
FLIR  Image  No.  1. 


4.4.1  Constant  Multiplier 

In  the  first  series  of  experiments,  the  edge  thresholds  were  altered  while  keeping 
the  multipliers  constant  to  determine  if  the  edge/texture  boundaries  do  indeed 
occur  where  the  analysis  of  the  histograms  indicates.  Figure  4-13  contains  three 
photographs,  each  with  a different  set  of  edge/texture  boundaries.  In  each  photo- 
graph, the  multipliers  for  each  group  were  kept  constant  with  the  following  values: 
groups  9 and  10  * X2,  groups  11  through  14  * X4,  and  group  15  * X8. 


Images  in  Figures  4-13b  and  4-13c  resulted  from  edge  thresholds  that  were  higher 
( thereby  creating  less  edge  terms)  than  the  analysis  in  paragraph  4.3.1  indicated. 
In  Figure  4-13a,  the  edge  threshold  was  set  in  accordance  with  the  study  in  4.3.1 
Seven  untrained  observers  were  asked  to  evaluate  three  separately  enhanced  images 
displayed  on  a CRT  monitor.  Of  the  three  images  in  Figure  4-13,  a consensus  pre- 
ferred the  edge  enhancement  of  image  b over  the  other  two. 


A second  series  of  experiments  was  conducted  on  the  test  imagery  to  evaluate  the 
effects  of  various  weighting  factors  for  each  group  of  edge  terms.  Figure  4-14 
contains  three  photographs,  each  with  a different  set  of  weighting  factors,  but 
with  identical  edge  thresholds.  The  edge  thresholds  for  image  b in  Figure  4-13 
were  used  for  these  experiments.  A majority  of  the  viewers  selected  the  image 
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Figure  4-13.  EFFECTS  OF  DIFFERENT  EDGE  THRESHOLDS 

labeled  b when  asked  to  grade  the  three  images  in  Figure  4-14  in  accordance  to  their 
preference . 


In  the  course  of  conducting  these  two  experiments,  two  undesirable  phenomena  were 
observed  on  the  CRT  monitor  that  are  not  very  discernible  on  the  photographs.  The 
first  phenomenon  was  a glittering  effect  on  the  monitor,  created  by  enhancement 
of  the  high-frequency  noise  (shot  noise)  prevalent  in  groups  14  and  15.  To  elimi- 
nate this  problem,  alteration  of  the  Haar  coefficients  in  groups  14  and  15  were 
inhibited.  Although  this  action  prevents  quite  a few  high-frequency  edge  terms 
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EFFECTS  OF  DIFFERENT  MULTIPLIERS 


The  second  undesired  phenomenon  observed  on  the  CRT  monitor  was  the  appearance  of 
numerous  horizontal  lines  where  a significant  horizontal  edge 
nal;  e.g.,  the  horizon  of  the  test 
the  interlace  format  of  the  TV  raster 


existed  in  the  origi 

image.  This  phenomenon  is  created  mainly  by 
scan.  Northrop' s sensor  processor  does  not 
store  a complete  field  of  video  data  and  then  operate  on  16  consecutive  lines  of 
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Figure  4-15.  EFFECTS  OF  SYSTEM  NOISE 


the  image.  Rather  it  computes  the  Haar  transform  on  the  16  consecutively  pre- 
sented TV  lines  of  a single  field  (every  other  line  of  the  scene).  Thus,  the  edge- 
enhancement  algorithm  interprets  a single  edge  in  the  horizontal  direction  as  being 
two  edges  (one  edge  per  field).  Over  a frame  time  when  these  two  edges  are  enhanced, 
multiple  horizontal  lines  appear  on  the  display.  This  problem  can  be  minimized  by 
using  weighting  factors  that  are  smaller  for  edges  in  the  horizontal  direction  than 
in  the  vertical  direction.  The  sensor  processor  has  the  capacity  to  separate  one 
group  into  two  in  its  present  form.  Group  11,  which  contributes  the  most  in  creat- 
ing the  undesirable  effects  of  horizontal  lines,  was  therefore  divided  into  two 
separate  groups  and  the  multiplier  for  the  half  of  group  11  that  measures  differences 
in  the  vertical  direction  was  reduced  from  X4  to  X2.  The  effects  of  this  action 
are  illustrated  in  Figure  4-16;  image  a is  a photograph  of  the  test  image  before 
group  11  was  separated,  and  image  b is  the  result  of  group  11  separation  and  reduc- 
tion of  the  multiplier  for  the  vertical  direction  differences. 

Im  summary,  the  parameters  of  the  constant  multiplier  algorithm  that  produced  the 
most  desired  effects  of  edge  enhancement  for  the  majority  of  the  observers  are  tabu- 
lated in  Table  4-5  and  illustrated  in  Figure  4-16b. 
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Figure  4-16.  EFFECTS  OF  TV  2:1  INTERFACE 
Table  4-5.  SELECTED  PARAMETERS  FOR  CONSTANT  MULTIPLIER 


Group  No. 

Amplitude  Values 

Multiplier 

From 

To 

9 

3 

12 

X2 

10 

3 

16 

X2 

1 1 (Vert . ) 

3 

20 

X2 

11  (Hor.) 

3 

20 

X4 

12 

2 

20 

X4 

13 

3 

20 

X4 

4.4.2  Variable  Multiplier 


The  majority  of  the  basic  parameters  that  affect  the  quality  of  edge  enhancement 
using  the  variable  multiplier  algorithm  have  been  determined  in  previous  experi- 
ments. The  remaining  question  to  be  addressed  for  this  algorithm  is  the  shape  and 
form  of  the  preferred  nonlinear  transfer  function.  Two  basic  forms  of  nonlinear 
transfer  functions  with  varying  degrees  of  slope  were  selected  for  study.  The  dif- 
ference in  the  two  basic  forms  is  that  in  one  transfer  function  the  maximum  output 
amplitude  was  curtailed  to  a smaller  value  than  the  other. 
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The  curves  in  Figure  4-17  for  nonlinear  transfer  function  No.  1 were  programed 
in  programmable  read-only  memories  (PROMS)  and  the  sensor  processor  modified  to 
allow  the  PROM,  rather  than  the  constant  multipliers,  to  alter  the  selected  edge 
terms . 

The  representative  test  image  was  subjected  to  varying  degrees  of  edge  enhancement 
by  selection  of  different  curves  for  the  various  groups  of  Haar  coefficients  to 
be  altered  and  subjectively  evaluated.  The  effects  of  this  is  shown  in  Figure  4-18 
along  with  a tabulation  of  curves  used  for  each  of  the  three  photographs.  It  is 
■shown  that  the  edges  of  the  targets,  trees,  cold  horizontal  band  across  the  center 
of  the  image  gets  progressively  sharper  and  discernible  from  image  a to  b to  c al- 
though the  artifacts  (shaped  like  Haar  functions)  are  also  increased.  The  majority 
of  the  evaluators  selected  image  b as  their  preference. 

The  procedures  and  tests  performed  on  nonlinear  transfer  function  No.  1 was  also 
conducted  on  the  curves  for  nonlinear  transfer  function  No.  2.  The  resiltant  effects 
on  the  display  (shown  in  Figure  4-19)  was  similar  to  the  test  conducted  for  transfer 
function  No.  2.  The  majority  of  the  evaluators  again  selected  image  b as  their 
preference . 

The  evaluators  were  not  able  vo  arrive  at  a consensus  in  selecting  the  preferred 
nonlinear  transfer  function,  because  the  resultant  effects  on  the  display  were  so 
similar.  As  a result,  the  curves  used  to  derive  an  enhanced  image  in  Figure  4-19b 
were  selected  for  further  testing. 

4.4.3  Descriptors 

The  Image  Descriptors  (ID's)  for  the  enhanced  versions  of  FLIR  Image  1 are  given 
in  Table  4-6,  which  shows  that  edge-enhanced  images  exhibit  substantially  larger 
edge  activities  with  only  a marginal  increase  in  texture  activity.  However,  some 
anomalies  do  exist  in  the  relative  levels  of  edge  activity  between  subjectively 
equivalent  images. 
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Figure  4-18.  VARIABLE  MULTIPLIER  ENHANCEMENT  - TRANSFER  FUNCTION  NO.  1 
4.5  FLIR  IMAGE  EXPERIMENTS 

Experiments  were  conducted  on  six  FLIR  images  in  order  to  evaluate  the  ability  of 
the  selected  edge  enhancement  algorithms  to  emphasize  edges  in  a wide  range  of 
diverse  images.  Each  image  was  subjected  to  the  three  enhancement  algorithms, 
selected  for  further  evaluation.  The  three  edge  enhancement  algorithms  are  (1)  the 
constant-multiplier  approach  using  the  fixed-ege  threshold  criterion,  (2)  the  vari- 
able multiplier  approach  using  the  fixed-edge  threshold  criterion,  and  (3)  the  vari 
able  multiplier  approach,  using  the  Image  Descriptor  (ID)  as  a criterion  for 
determinirg  edges  in  the  scene. 


Figure  4-19.  VARIABLE  MULTIPLIER  ENHANCEMENT  - TRANSFER  FUNCTION  NO.  2 

Photographs  of  these  experiments  are  shown  in  Figures  4-20  through  4-25.  Descrip- 
tors of  selected  enhancement  algorithms  are  listed  in  Table  4-7. 

Experiments  for  the  edge-enhancement  algorithm  using  the  image  descriptor  were  con- 
ducted on  the  sensor  processor  in  conjunction  with  the 'instrumentation  system  (de- 
picted in  Figure  4-1).  The  instrumentation  system,  with  its  general  purpose  com- 
puter, was  used  to  implement  the  complex  algorithm  of  the  image  descriptor.  Since 
the  instrumentation  system  cannot  process  video  in  real  time,  each  rectangular  area 
(containing  12  sectors)  processed  by  the  ID  enhancement  algorithm  was  integrated 
over  eight  runs.  This  was  performed  to  simulate  the  effects  of  integration  by  the 


Table  4-6.  DESCRIPTORS  FOR  IMAGE  1 


Figure 

Tone 

Edge  Activity 

Texture  Activity 

2-13a 

19 

8883 

2619 

2-13b 

22 

5551 

559 

2-13c 

22 

2324 

475 

2-14a 

22 

937 

408 

2-14b 

23 

5859 

648 

U 

1 
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22 

12,700 

1459 

2-15a 

22 

6050 
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2-15b 

22 

1094 

596 

2- 16a 

22 
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599 

2-16b 

22 

779 
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2- 18a 

22 
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389 

2-18b 

22 

913 

500 

0 

00 

1 

CM 

21 

1107 

563 

2- 19a 

22 

656 

392 

2-19b 

23 

975 

507 

2-19c 

22 

1081 

562 

CRT  display  and  eye-brain  combination  upon  slightly  varying  video  signals  produced 
by  the  video  camera. 

In  addition  to  the  experiments  conducted  on  the  six  FLIR  images,  identical  experi- 
ments for  constant-multiplier  and  variable-multiplier  approaches  were  conducted 
on  a live  parking-lot  scene  (not  a phototransparency)  using  the  vidicon  camera 
(visual  spectrum)  shown  in  Figure  4-26.  It  can  be  readily  seen  that  the  edge- 
enhanced  images  of  the  parking  lot  were  significantly  improved,  compared  to  the 
edge-enhanced  images  of  the  FLIR's  photographic  transparencies.  It  was  deduced 
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Figure  4-22.  IMAGE  6 
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Table  4-7.  DESCRIPTORS  OF  SELECTED  ENHANCEMENT  ALGORITHMS 


Figure 

Tone 

Edge  Activity 

Texture  Activity 
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Table  4-7 . DESCRIPTORS  OF  SELECTED  ENHANCEMENT  ALGORITHMS  (Continued) 

Figure 

Tone 

Edge  Activity 

Texture  Activity 

4 -25a 

19 

516 

263 

4-25b 

19 

665 

350 

4-25c 

19 

739 

425 

4-25d,e,  f Orig 

27 

589 

326 

4-25d,e,f  Enh 

27 

1 141 

595 

that  the  combination  of  smaller  dynamic  range  (reduced  gray  shade  resolution)  and 
lower  Modulation  Transfer  Function  (MTF)  of  FLIR  transparencies  contributed  to  the 
difference  in  the  performance  of  the  edge-enhancement  algorithms  on  the  two  types 
of  test  images.  These  two  system  parameters  (dynamic  range  and  MTF)  have  a great 
impact  on  the  performance  of  the  edge-enhancement  algorithm  since  it  determines 
the  algorithm's  ability  to  separate  edges  from  texture  information  and  noise;  i.e., 
if  these  two  system  parameters  are  low,  the  smaller  edge  terms  will  not  be  distin- 
guishable from  texture  or  noise  terms. 


Seven  observers  were  seated  approximately  30  inches  away  from  an  11-inch  CRT  monitor 
and  shown  the  results  of  the  selected  edge-enhancement  algorithms  for  all  six  FLIR 
images.  On  the  average,  five  of  seven  of  the  evaluators  selected  the  image  created 
by  the  variable  multiplier  algorithm  (b)  as  their  preference.  However,  preferences 
for  specific  images  by  individual  evaluators  varied. 

4-6.  CONCLUSIONS 

The  image  descriptor  based  on  the  edge  criteria  described  in  the  first  quarterly 
report  provides  a measure  for  edges.  In  order  to  meet  the  goal  of  simple  process- 
ing at  real-time  video  rates,  the  fixed  edge  threshold  algorithm  is  recommended 
for  the  extraction  of  edges.  The  variable  multiplier  algorithm  is  recommended  for 
edge  emphasis. 


The  recommended  algorithms  are  somewhat  scene  dependent.  The  dynamic  range,  con- 
trast, and  amount  of  activity  influence  the  effectiveness  of  the  edge  enhancement 
to  a degree.  At  this  time,  we  feel  that  dynamic  range  compression  (gray-scale 
stretch),  brightness,  and  gain  control  performed  before  edge  enhancement  will  miti- 
gate this  problem.  Unfortunately,  this  could  not  be  included  within  the  scope  of 
this  experiment,  but  would  be  an  interesting  continuation  in  the  future. 


These  algorithms,  which  indicate  a high  level  of  noise  immunity,  are  basically  im- 
pervious to  low-frequency  noise.  Random  high-frequency  noise  tends  to  be  accentu- 
ated in  an  edge-enhanced  picture,  but  on  the  whole,  it  should  not  interfere  with 
target  recognition.  Noise  caused  by  scan-interlace  processing  can  be  mitigated  by 
moderating  the  horizontal  edge  Haar  coefficients. 
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Finally,  it  is  felt  that  the  edge  enhancement  techniques  could  be  built  into  a real- 
time 2D  Haar  image  processor  as  an  integral  part  of  a next  generation  or  upgraded 
common  module  FLIR  using  presently  available  technology. 


SECTION  5 

ADAPTIVE  SIGNAL  PROCESSING 


5.1  INTRODUCTION 

The  following  discussion  covers  dc  restoration  and  dynamic  range  compression.  Under 
dc  restoration,  the  desirable  and  undesirable  effects  of  ac  coupling  in  FLIR's  are 
discussed.  These  effects  include: 

a.  Exponential  decay,  or  droop 

b.  Ability  to  distinguish  small  targets  against  low-contrast  backgrounds 

c.  Drift  in  average  waveform  level 

d.  No  Automatic  Gain  Control  (AGC)  capability  to  maintain  good  contrast  tor 
all  signal  levels. 


This  discussion  investigates  Haar-transform-besed  algorithms  designed  to  correct 
effects  a,  c,  and  d,  and  to  improve  the  tradeoff  between  a and  b. 


Dynamic  range  compression  involves  nonlinear  methods  for  optimizing  use  of  the  avail- 
able gray  scale.  This  discussion  outlines  an  equivalent  approach  that  rearranges 
the  nonlinear  operations  in  a manner  better  suited  to  Haar  transform  processing. 


5.2  DC  RESTORATION  PROBLEM 


5.2.1  Problem  Definition 


AC-coupled  electronics  is  generally  characterized  by  exponential  decays  toward  zero, 
as  in  Figure  5-1.  This  figure  typifies  Infrared  (IR)  line  scans  of  uniform  hot 
and  cold  objects,  which  largely  fill  the  field  of  view.  The  illustrated  decays 
result  from  an  ac  coupling  that  blocks  all  frequencies  (f)  below  some  cutoff 
point  f - (coupler  RC  time  constant)  Larger  fQ  values  produce  greater  droop. 
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Figure  5-1.  AC  COUPLING  EFFECTS  IN  PARALLEL  SCAN  FLIRS 


They  also  provide  a larger  stop  band  for  preventing  the  passage  of  1/f  noise. 
Recent  studies^,  however,  claim  that  l/f-noise  immunity  can  be  achieved  with  rela- 
tively small  fQ,  producing  negligible  droop  per  scan  time  (T). 

These  investigations  consequently  disregard  single-scan  droop  effects.  They  do, 
however,  address  the  problem  of  cumulative  droop  over  many  scans.  Cold  bars  repre 
sent  one  method  of  coping  with  cumulative  or  multiscan  droop,  but  their  operation 


*Tack,  D.  H. , and  P.  Narendra,  "Automated  Image  Enhancement  Techniques,  2nd  Gen 
eration  FLIR"  Report  #DAA653-76-C-0195 , Honeywell,  Inc.,  Systems  Research  Cen- 
ter, 1976. 
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sometimes  leads  to  saturation  and/or  inefficient  use  of  the  available  gray  scale 
2 

of  the  display.  Present  studies  therefore  seek  methods  to  eliminate  multiple-scan 
droop  via  Haar  transform  processing  without  recourse  to  cold  bars. 

One  consequence  of  this  approach  to  droop  elimination  is  improved  vertical  corre- 
lation. Such  improvement  would  relieve  the  type  of  problems  characterized  by  Fig- 
ure 5-2. 
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Figure  5-2.  VERTICAL  CORRELATION  LOSS  IN  PARALLEL  SCAN  FLIRS 

This  figure  shows  how  a hot  cencral  object  on  a cold  background  degenerates  into 
cooler  image  against  a distorted  background.  Parallel  scans  AA  and  BB  each  produce 
zero-average  outputs  by  virtue  of  the  ac  coupling.  Unlike  AA,  however,  scan  BB 
can  only  achieve  this  result  through  deliberate  distortion  of  its  dc  level  (and 
hence,  the  background). 


Other  goals  include  preservation  and  enhancement  of  the  desirable  effects  of  ac 
coupling.  Straight  dc  coupling  gives  relatively  little  play  to  small  amplitude 
signals.  The  solid-line  waveform  in  Figure  5-3,  for  example,  might  represent  a 
match  in  front  of  a blast  furnace. 

The  central  pip  may  not  span  even  one  gray  level  in  the  output  display,  and  the 
match  may  go  undetected.  Ideal  ac  coupling,  on  the  other  hand,  would  assign  larger 
spectral  gains  to  signal  components  with  a more  spike-like  appearance,  as  depicted 
by  the  broken-line  curve.  The  match  would  now  stand  out  unmistakeably  from  the 
blast  furnace  despite  the  small  contrast. 


With  the  cold  bar,  extremely  hot  or  cold  scenes  push  the  horizontal  axis  in  Fig- 
ure 5-1  to  either  end  of  the  available  gray  scale — possibly  beyond. 
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Figure  5-3.  IDEAL  DC-  AND  AC-COUPLED  INFRARED  IMAGING 

Actual  ac  coupling  (via  an  RC  filter)  cannot  achieve  quite  so  perfect  a result  aa 
the  broken-line  curve.  The  blast  furnace  profile  would  distort  in  aaich  the  same 
way  as  the  hot  object  in  Figure  5-1.  The  superimposed  match  waveform  meanwhile 
will  grow  in  sice  and  will  acquire  an  exponential  tail.  In  short,  typical  ac  cou- 
plings improve  pictorial  contrast  at  the  cost  of  skewing  the  iauige. 

These  descriptions  can  be  verified  by  convolving  the  solid-line  waveform  of  Figure 
5-3  with  the  RC  filter  weighting  function  of  Figure  5-4. 
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Figure  5-4.  WEIGHTING  FUNCTION  OF  AC  COUPLING  CIRCUIT 

The  negative  area  under  the  exponential  portion  of  the  curve  exactly  balances  the 
unit  positive  area  under  6(t),  indicating  that  the  dc  component  of  any  input  wave- 
form will  be  totally  blocked.  The  6-portion  reproduces  the  input  waveform  exactly, 
while  the  exponential  tail  introduces  drooping.  Large  T flattens  this  tail  and 
minimises  the  droop  while  simultaneously  lowering  the  cutoff  frequency.  The  pres- 
ent case  assumes  sufficiently  large  T so  that  droop  is  minisMl  within  a single-framt 
interval,  and  only  becosws  a problem  when  viewed  over  multiple  frames.  This  choice 
of  T,  however,  minimizes  the  differentiation  properties  of  the  circuit  and  reduces 
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the  contrast  between  natch  and  blast  furnace.  The  present  study,  therefore,  seeks 
to  use  Hear  transforms  for  producing  high-contrast  images  while  removing  objection- 
able droop  effects. 


' 


A third  goal  concerns  the  ability  of  Haar-aided  FLIR's  to  maintain  average  signal 
level  at  the  middle  of  the  display  gray  scale.  Without  this  ability,  sudden 
changes  in  sverage  scene  brightness  can  saturate  the  FLIR.  The  other  extreme, 
however,  leads  to  complete  loss  of  scene  brightness  information.  This  area  of 
study,  therefore,  aims  at  a flexible  intermediate  level  of  control. 

A fourth  goal  would  uniformly  stretch  or  compress  the  signal  waveform  to  improve 
gray  level  use.  The  broken-line  waveform  of  Figure  5-3,  for  example,  might  be  ver- 
tically stretched  (accentuated)  until  the  tip  of  the  match  coincided  with  the  upper- 
most  gray  level.  Such  a nonlinear  operation,  however,  would  depend  upon  minimum 
and  stfximum  values  of  the  display  intensity,  and  finds  no  solution  via  Haar  trans- 
forms. This  study,  therefore,  considers  approximate  linear  solutions  suited  to 
Haar  transforms. 

These  four  goals, 

a Compensating  multiscan  droop 

a Accentuating  the  match  in  front  of  the  blast  furnace 
a Adjusting  average  signal  level  to  midgray  display  level 
a Adjusting  ac  signal  amplitude  in  relation  to  total  span  of  gray  scale, 

represent  the  four  aspects  of  the  dc  restoration  problem  addressed  here. 

5.2.2  DC  Restoration  Solution 

5. 2. 2.1  Multi  scan  Droop  Solution  - This  section  first  provides  a baaic  approach 
to  solving  this  problem.  It  then  describes  the  impact  of  stability  considerations, 
as  well  as  a method  for  maintaining  transient  (absolute)  stability.  The  section 
further  details  the  sensitivity  of  the  correction  scheme  to  fluctations  in  coupling 
network  parameters.  It  presents  the  results  of  a computer  simulation  of  the 


corrective  network  in  suppressing  ac  coupling  circuit  droop  effects.  Finally,  the 


•action  provide*  a quantitative  expression  for  a basic  design  parameter,  K,  of  the 
corrective  network.  (An  interned iate  step  involves  the  derivation  of  subsidiary 
constant  I,  which  is  also  used  in  the  computer  s iwula t ion.) 

5. 2. 2. 1.1  Basic  Approach  - Figure  5-5  characterises  the  noninal  dc  level  of  a 


5. 2. 2. 1.1  Basic  Approach  - Figure  5-5  characterises  the  noninal  dc  level  of  a 
linear  scene  that  renains  fixed  over  scans  1 through  5 and  then  changes  abruptly. 
Such  a linear  scene  would  correspond  to  the  output  fron  one  parallel-scan  FL1K 
photodetector.  Higher  hamonics/Haar  sequenciea  of  each  scan  wavefom  are  assuawd 
to  pass  through  the  sc  coupling  circuit  relatively  unaffected.^  Multiscan  droop, 
therefore,  appears  to  impact  primarily  on  the  dc  levels  of  successive  scans  as  in 
Figure  5-5. 


DC  LEVELS  OF  INDIVIDUAL  SCANS  AT 
INPUT  TO  COUPLING  NETWOAK  . 
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Figure  5-5.  MULTISCAM  DROOP  CHARACTERIZATION 

The  dc  Hear  term  of  scan  1 will  assume  an  amplitude  of  Xj,  scan  2 yields  x^  “ 
Xjexp(-t/T),  scan  3 produces  x^  ■ Xjexp(-2t/T) , etc.,  where  T ■ network  time  constant. 
Such  amplitudes  represent  sample  values  of  the  network  response  to  the  input 


Scan  waveform  harmonics  suffer  relatively  little  attenuation,  particularly  when 
the  sc  coupling  calls  for  large  time  constants  and  low  cutoff  frequencies.  All  Haar 
terms  contain  soaw  spectral  components  below  cutoff,  but  the  proportion  becoaws 
progressively  smaller  with  increasing  sequency. 
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t 


n 

i; 


i i 


l 

i 


function  XjU(t);  u(t)  ■ unit  atop.  In  Urai  of  Laplace  trana forma,  they  can  be 
characteriaad  by  the  tranafer  function 


A(a) 


Ta 

1 ♦ Ta 


auggeating  a corrective  network  baaed  on 


B(e)  - A‘l(a) 


1 ♦ 


1_ 

Ta 


b(t) 


6(t) 


♦ 


u(t) 

T 


In  teraa  of  the  aaaipled-data  model,  weighting  function  b(t)  effectively  requires 

that  the  corrective  network  (1)  take  the  current  value  x of  the  diatorted  waveform, 

n 

and  (2)  add  to  it  I • (Sail  previoua  value*,  xr_^ , xn_j  ...). 


Thia  correaponda  to  diacrete  operator 


b(a)  - 1 ♦ R £ a"  • 1 ♦ K“ 


n*l 


TT^-i T 


a - Delay  of  d aeconda,  equal  to  the  aample  tin 
-d/T 


K - 1 - « 


(i 


>h  5. 2. 2. 1.4) 


which  can  be  integrated  into  the  general  Haar-proceaaed  FLIR  according  to  the  algo- 
rithai  of  Figure  5-6.  The  aaate  algorithm  can  alao  be  applied  to  the  next  higher 
Hear  aequenciea,  but  the  aoMll  increaae  in  picture  fidelity  may  not  warrant  the 
added  complication.  Thia  procedure  ahould  effectively  counter  the  loet  horiaon 
problem  ae  well  aa  the  diatorting  phenomenon  of  Figure  5-2. 

5. 2. 2. 1.2  Stability  Conaiderationa  - The  preceding  diacuaaion  haa  preaented  the 
baaic  corrective  procedure  for  the  dc  reatoration  problem.  Stability  conaidera- 
tiona however  etill  remain  to  be  inveatigated.  Theae  include  both  the  abaolute 
atability  of  the  corrective  network  and  ita  parametric  behavior;  i.e.,  ita  ability 
to  accomodate  parameter  changea  in  the  original  ac  coupling. 


5-7 


condition*.  This  appears  superimposed  on  the  steady-state  response  of  the  filter. 
Absolutely  stable  filters  generally  produce  exponentially  decreasing  transients 
which  quickly  reduce  to  negligible  proportions  compared  to  the  steady-state  response. 
The  increasing  transients  of  unstable  filters,  on  the  other  hand,  often  arise  from 
unwanted  positive  feedback  involving  active  elements  within  the  filter.  In  the 
Laplace  transform  domain,  stable  and  unstable  transients  correspond  to  poles  in 
the  right  (positive)  and  left  (negative)  halves  of  the  complex  s-plane  respectively. 

Proposed  filter  B(s)  contains  a pole  at  the  origin  of  the  complex  s-plane,  and 

represents  a potentially  unstable  condition.  One  corrective  approach  using 

. . . . . . 4 

pseudo-inversion  was  investigated,  but  does  not  seem  particularly  promising.  For 

this  reason,  an  alternate  approach  was  adopted,  based  on  the  following  pragmatic 
conditions: 


| 


4 -I 

Pseudo-inversion  replaces  the  original  specification  B(s)  * A (s)  by  the  more 

reasonable  condition 

jCO 

J |B(s)  A(s)  — 1 1 ^ dm  —►Minimum 
-j® 

This  minimum  is  in  turn  constrained  by  the  requirement  that  the  noise  output  of 

the  system 

f 2 

/ | B (s ) | ds  ” Nq;  ■ Input  white-noise  spectral  density 

-J- 

holds  to  a low  fixed  value.  Analysis  based  on  variational  calculus  effectively 
replaces  the  above  pole  by  two  weaker  ones  in  the  positive  half  o):  the  s-plane 
(unstable)  and  in  the  negative  half  (stable).  Synthesis  calls  for  deleting  the 
fraction  term  representing  the  positive  pole,  and  does  indeed  produce  a 
stable  filter.  Its  corrective  capabilities  however  seem  weak,  and  the  approach 
was  not  pursued  further. 


■ 

i 
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The  object  of  the  high-pass  filter  in  FLIR  systems  is  to  emphasise  the  presence 
of  sawll  targets,  especially  when  they  appear  against  a low-contrast  background 
(this  is  discussed  extensively  in  paragraph  5. 2. 2. 2 of  this  report).  Since  small 
targets  occupy  only  sbmII  portions  of  the  scan  interval,  they  generate  relatively 
high-frequency  components  in  the  photodetector  output.  The  filter  seeks  to  empha- 
sise these  components  by  suppressing  the  low  frequencies.  It  achieves  this  by  rol- 
ling off  the  low-frequency  response  at  10  dB  per  decade  of  frequency  decrease 
below  break-frequency  fj-  Below  a certain  frequency  fj,  however,  progressive 
attenuation  has  reached  such  an  extreme  state  that  further  roll-off  is  not  needed. 

A further  break- frequency  can  therefore  be  inserted  at  fj  which  flattens  the  response 
for  all  f<fr 


Thus,  a conservative  form  of  high-pass  filter  would  possess  the  transfer  function 


sew. 


A’(s)  - 


1 


S+OJU 


W1  - 2"fl 


t*>2  * 2wf2 


Note  that  A'(s)  corresponds  to  the  physically  realisable  rc  network  of  Figure  5-7. 
This  network  differs  from  the  original  high-pass  filter  only  in  the  addition  of 
resistor  rj.  It  provides  a more  reasonable  model  since  non-sero  r^  will  always 
shunt  C,  if  for  no  other  reason  than  the  finite  Q of  the  capacitor.  The  polarized 
capacitors  used  in  FLIR  systems  (typically  about  33  microfarads)  suffer  a dissipa- 
tion specified  by  K (between  0.01  and  0.1,  depending  upon  temperature)  in  the 
relation 


I - KCV 

Thus,  KC  specifies  the  equivalent  parallel  conductance  and  leads  to  the  result 
Wj  * K ~ 0.063 
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r 


See  paragraph  5. 2. 2. 1.5  for  determination  of  constant  multiplying  E.  Figure  5-8 
gives  the  corresponding  algorithm  for  the  Haar-transfora  processor- 


4*0710 


Figure  5-8.  PROCESSOR  ALGORITHM  FOR  ABSOLUTELY 
STABLE  CORRECTIVE  NETWORK 


5. 2. 2. 1.2. 2 Parametric  Behavior  - Assuming  absolute  stability,  potential  problems 
can  still  arise  if  the  corrective  network  uses  inaccurate  numerical  values  for  ujj  and 
Hg.  Such  inaccuracies  might  be  caused  by  temperature  variations  in  the  ac  coupling 
or  by  any  other  form  of  parameter  drift.  In  particular,  suppose  that 

A(s)  - S+“l  **“l 

+ Au>  j 

B(s)  - 

The  impulse  response  of  the  tandem  combination  changes  from  an  ideal  value  of  6 ( t ) 
(i.e.,  perfect  correction)  to 

6(t)  * AujjC  wlt  - Au>2  « **2  C 

Parameter  drift  may  therefore  introduce  some  performance  degradation  but  it  will 
not  produce  instability.  This  degradation  can  be  expressed  in  terms  of  the  mean 
square  transient  error 


i2irW 


-i2wW 

W * Signal  bandwidth  in  Hz 
E(s)  ” Laplace  transform  of  input  signal 


1 


\ 
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To  simplify  calculation,  take  E(s)-1,  indicating  a flat  power  spectrum  of  unit 
strength  inside  the  (0,u>)  Ha  bandwidth,  and  zero  strength  outside.  Substitution  of 
the  preceding  expressions  for  A"  and  B'  then  shows  that 


tan  * 


ft) 


(Au*2  ) tan 


-(ft) 


This  expression  attains  a maximum  value  of 


/ow)2  ,»2\ 

5 \—  * — / 


for  arbitrarily  large  W. 

Thus,  for  Wj  m 0. 063 , u>2  * 25.133,  Aoj^  * u< ^ / 10  and  = 10 , 

an  input  power  of 

VW1 

(1  watt/Ha)  ( — — -)  = 3.99  watts 

2tt 

produces  a waveform  distortion  component  with  a power  or  0.126  watts.  This  suggests 
signal/distortion  ratios  as  high  as  15  dB  even  for  a 10  percent  error  in  network 
design  parameters. 

5. 2. 2. 1.3  Simulation  of  Correction  Algorithm  - A computer  simulation  of  the 
original  corrective  technique  based  on  b(z)  has  been  successfully  applied.  In  the 
following  examples,  sample  time  d (A  in  the  computer  printout)  has  been  equated  to 
the  interval  between  successive  computations  of  Haar  terms  for  the  same  16-by-16 
pel  block: 


(30  frames/sec ) 


0.03333  sec 


Actually,' W * )/2ir  in  the  expression  for  M.  The  limiting  expression 

above  however  gives  a clearer  indication  of  the  effects  of  parameter  drift. 


This  figure  is  typical  of  parallel-scan  FLIRs . These  examples  also  employ  a nominal 
time  constant  of 


1 

T = ^ u u ^ = 0-0398  sec 

2 it  (4  Hz ) 

Example  1 in  Table  5-1  assumes  perfect  agreement  between  the  time  constant  of  the 
high-pass  filter  and  the  corresponding  K-value  of  the  corrective  network.  Column  X 
represents  a step-function  type  waveform  input  to  the  high-pass  filter  from  the 
FLIR  photodetector;  Y represents  the  filter  output  and  input  to  the  corrective  net- 
work; Z gives  the  output  of  the  corrective  network.  Note  that  Z represents  a per- 
fect restoration  of  X.^ 

Example  2 repeats  the  same  general  X-Y-Z  scenario,  but  assumes  that  the  high-pass 
filter  time  constant  has  drifted  by  1 percent  from  its  nominal  value.  Example  3 
assumes  5 percent  drift  and  example  4 assumes  10  percent. 

The  computer  program  permits  the  simulation  of  five  examples  per  run,  though  only 
four  have  been  used  here.  Note  that  corrected  output  Z shows  progressively  poorer 
fidelity  for  increasing  mismatch  in  T.  For  10  percent  mismatch  (T=0. 00438),  a 
typical  error  of  6.43  percent  over  the  bandwidth  between  u>j and  (3.99  Hz)  trans- 
lates to  an  equivalent  M-value  of  (0.0643)^  x (3.99)  = 0.0165,  or  about  18  dB 

signal/distortion  ratio.  This  approximate  calculation  seems  in  rough  agreement  with 
the  theoretically  derived  15  dB  figure  cited  earlier. 

5. 2. 2. 1.4  Determination  of  Constant  K - This  determination  first  characterizes 
the  sampled-data  equivalent  of  the  RC  high-pass  filter  A(s).  It  then  examines  the 
manner  in  which  the  train  of  ouput  samples  might  drive  a corrective  network.  In 
particular,  it  looks  for  the  particular  value  of  K which  allows  the  corrective  net- 
work to  recreate  the  original  sampled  data  input  to  A(s). 


. 


- 

Because  of  the  finite  time  interval  over  which  X is  specified  and  the  exclusion 
of  noise,  questions  of  stability  do  not  arise. 

L 
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To  this  end,  consider  the  inverse  Laplace  transform  of  A(s):  i.e.,  the  impulse 
response  of  the  high-pass  filter: 


i(t)  = bit)  - (1/T)  t 


(5-1) 


(5-2) 


Expression  a(t)  indicates  that  the  sampled  filter  output  yQ  should  represent  the 
sum  of  the  present  input  sample  xq  and  a scaled,  exponentially  attenuated  train  of 
previous  input  samples  *n_j»  xn_2*  ‘ * * Specifically, 


-R  (x 
n \ n-1 


-d/T  -2d/T 

< + x t 

n-2 


(5-3) 


constant  R*l/T,  as  (5-2)  might  seem  to  suggest.  It  does,  however,  approach  (d/T) 
as  d approaches  zero. 

To  determine  the  correct  value  of  R,  take  all  x =1,  sum  the  resulting  geometric 

m 

series  within  the  brackets  and  obtain 


R/-d/T  _ ; d/T\ 
\ l-Id/T  / 


(5-4) 


Since  Yn  approaches  zero  as  n becomes  arbitrarily  large, 


d/T 

R = c ' -1 


Substitution  of  this  value  into  the  general  expression  yields 


-(n-1 ) d/T 


(5-5) 


(5-6) 


These  samples  constitute  the  driving  function  for  a corrective  network  character- 


ized by 


b(z ) « 1 + K £ zn 


(5-7) 


Network  output  corresponds  to 
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* K (yr-l  * yr-2  * 


y ♦ 

r-3 
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(5-8) 


Replacing  the  bracketed  y's  by  their  exponential  equivalents  in  (5-6)  and  summing 
the  resulting  geometric  series, 


-(r-1 )d/T 


/,  K \ 
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(5-9) 


The  last  fraction  in  5-9  represents  wf  for  arbitrarily  large  r,  and  should  equal 
= 1 for  perfect  correction.  This  yields 


Note  that  substitution  of  this  value  into  (5-9)  renders  w^  = 1 not  only  for  large 
values  of  r but  for  all  values. 


5. 2. 2. 1.5  Determination  of  Constant  in  b1  (z)  - The  same  rationale  which  led  to 

(5-3)  also  applies  to  the  absolutely  stable  network  characterized  by  b'  (z).  Taking 
y^  and  w^  as  the  input  and  output  sequences  of  sampled  values, 
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where 


> - « 


-u),  d 


and  Q is  the  constant  to  be  determined.  From  (5-6) 


r(n-l)  d/T  n-1 
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p - rd/T 


(5-11) 
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(5-13) 
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Substituting  (5-12)  into  (5-10)  and  sunning  the  geometric  series  within  the 
bracketed  tern, 


P°-1  (i  ♦ ax  \ . 9i? 

\ P-Y  / p-Y 


(5-14) 


This  expression  gives  a corrective  network  output  which  attempts  to  restore  the 
train  of  unit  pulses  which  characterized  the  original  input  to  the  high-pass  filter. 
To  the  extent  that  a is  slightly  less  than  unity,  it  can  only  approximate  this 
result.  Note,  however,  that,  with  Y temporarily  taken  equal  to  unity,  the  first 
term  in  (5-14)  varies  with  n while  the  second  term  gives  the  desired  constant  condi- 
tion over  all  n.  Thus,  W * constant  if  the  coefficient  of  pn  * is  zero;  i.e., 

n 


(u)  -w  ) 

1 - (p/Y)  - l-< 2 1 


This  yields  the  coefficient  used  in  the  expression  for  b'  (z) 


5. 2. 2. 2 Match  - dlast  Furnace  Contrast  Solution 


(5-15) 


Large  time  constants  tend  to 
suppress  this  type  of  contrast.  Assuming  the  droop  problem  to  be  adequately  cor- 
rected by  the  algorithm  in  Figure  5-6,  contrast  can  be  reinstated  by  boosting  the 
values  of  the  higher  sequency  Haar  coefficients.  Such  a process  may  in  many  ways 
resemble  a new  ac  coupling  with  reduced  T,  but  it  does  not  reintroduce  droop. 

One  algorithm  that  achieves  this  goal  would  increase  all  Haar  coefficients  of  the 
droop-corrected  image  in  proportion  to  their  sequencies.  Thus,  a 2D  image: 

I(x,y)  * £ £ afma  h(r,m,.<)h(p,n,y) 
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would  convert  to 


J(x,y)  ■ C V V (r+l)(p+l)  a a h(r ,m,x)h(p,n,y) 
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where  C ■ any  normalization  constant  consistent  with  the  limited  display  gray 
scale.  Note  that  h(-l,l,x)  represents  the  dc  term.  Other  algorithms  might  replace 
(r*l)(p+-l)  by  (r+1)8  (p+l)1  where  the  exponents  might  be  empirically  derived  for  the 
class  of  scenes  under  consideration. 

5. 2. 2. 3 Gray-Scale  Centering  Solution  - Perfect  centering  can  be  achieved  by 
replacing  dc  coefficient  a . . . . by  whatever  value  corresponds  to  the  middle 

of  the  display  gray  scale.  This  process,  however,  would  disregard  any  significant 
changes  in  scene  brightness.  In  removing  this  difficulty,  the  algorithm  of  Fig- 
ure  5-9  converts  a . . . . to  (midgray  value)  + m • a , , where  m - some 

"i  , 1 ,”1  , I “MM  * I * 

small  positive  constant. 


This  scheme,  however,  would  respond  to  the  brightness  levels  of  individual  16-by-16 
pel  blocks  within  the  scene,  without  any  regard  to  smoothing  the  transition  from 
one  block  to  its  neighbors.  Such  discontinuities  result  in  the  appearance  of 
artifacts . 

To  eliminate  these  artifacts,  an  improved  correction  scheme  manipulates  only  the  ' 1 

average  value  of  all  individual-block  brightness  levels.  For  each  block,  the  -» 

deviation  between  individual  brightness  level  and  the  iverall  average  is  preserved 
as  a fixed  value.  Manipulations  of  the  average  value  then  will  change  overall 
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scene  brightness,  but  will  not  affect  the  relative  variation  of  brightness  from 
block  to  block.  The  result  will  be  a brightness  control  scheme  free  of  artifacts. 
Pigure  5-10  outlines  the  basic  process.  Successive  dc  Haar  terms  are  stored  in  the 
left-hand  shift  register  just  long  enough  to  reduce  each  one  by  their  common  aver- 
age value.  Each  processed  term  is  then  increased  by  A voltage  corresponding  to 
mid-gray,  with  a small  fraction  (m)  of  the  average  superimposed.  Read-out  to  the 
inverse  2D  Haar  transformer  proceeds  through  a second  shift  register  which  preserves 
the  original  order  in  which  the  dc  terms  were  read  in.  The  higher  order  terms, 
meanwhile,  undergo  a compensating  delay;  those  terms  corresponding  to  a particular 
16-by-16  pel  block  arrive  at  the  inverse  transformer  at  the  same  time  as  their 
associated  a . . . . term. 

5. 2. 2. 4 AC  AMPLITUDE  STRETCHING  SOLUTION 

Amplitude  stretching,  or  contrast  control,  is  a basically  nonlinear  process.  It 
adjusts  the  ac  gain  of  image  waveform  g(x,y)  such  that 


g(x,y)-g(average) I 


* Half-span  of  display  gray  scale 


Certain  approximations,  however,  can  linearize  this  process,  adapting  it  to  Haar- 
traneform  processing.  The  rationale  for  these  approximat ions  follows  from  Figure 
5-11.  This  figure  assumes  a normal  distribution  of  intensity  deviations  about  mid- 
gray.^ As  Ax  increases,  a larger  total  interval  of  gray  scale  deviations  will  be 
covered,  but  the  average  value  associated  with  each  gray  level  tends  to  decrease, 
(i.e.,  each  gray  level  zone  tends  to  slide  down  the  Gaussian  curve  to  one  side  or 
the  other,  with  increasing  x).  Quantitatively  expressed, 


Expected  Information  of  Display 


E < 


Standard  Deviation 


Intensity  deviations  of  a given  scene  may  or  may  not  follow  a normal  distribution. 
Averaged  over  the  many  scenes  that  might  apply  to  a given  scenario,  however,  the 
central  limit  theorem  suggests  a progressive  approach  to  the  normal  model. 
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Figure  5-11.  CONTRAST  LINEARIZATION  ANALYTICAL  MODEL 


will  reach  a maximum  for  some  appropriate  value  of  x.  Approximate  calculations 
show  that 

tttx  (Optimum  Gray-Scale  Span)  = z 

particularly  for  N greater  than  about  10.  In  effect,  gain  adjustments  should  not 
force  the  gray  levels  to  cover  the  total  span  of  fluctuations  but  only  about  1.5 
times  the  standard  deviation  of  all  the  expected  scenes.  Fluctuations  that  fall 
below  level  1 or  above  N may  be  lost  because  of  saturation,  but  this  sacrifice  en- 
hances gray-level  resolution  and  maximizes  overall  display  information. 

Following  is  the  approximate  calculation  used  to  derive  the  optimizing  condition: 
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where  the  limit  value  applies  to  large  N. 

Haar  implementation  of  these  principles  associates  activity  with  standard  devia- 
tion, i.e., 

«r2  - Activity  - £ £ a2 

r,m  p,n 

Figure  5-12  outlines  the  general  algorithm 
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Figure  5-12.  HAAR  CONTRAST  CONTROL 


This  algorithm  takes  V as  the  voltage  span  between  upper  and  lower  saturation 

limits.  Thus,  with  N gray  levels  in  the  display,  (V  )/N  represents  the  incre- 

sat 

mental  voltage  needed  to  move  from  one  gray  level  to  the  next.  This  voltage  should 
produce  a display  intensity  change  of  Ax,  indicating  a required  volts-per-unit- 
intenaity-change  conversion  gain  of 

V V 

sat  ■ sat 

NAx  nyfl 


In  actuality,  however,  the  conversion  gain  for  a given  FLIR  display  may  correspond 
to  some  fixed  paraiseter  a (volts  per  unit  intensity  change).  A variable  gain  (adapt- 
ive gain)  multiplier  Y is  therefore  needed  to  make  ud  the  diacrenanrv: 


Note  that  relatively  active  scenes  with  large  intensity  fluctuations  in  gray  shades 
correspond  to  large  <r  which iin  turn  invoke  small  y.  In  this  instance,  adaptive  Haar 
Contrast  Control  serves  as  a gray-shade  compressor.  Similar  reasoning  for  quiet 
scenes  (low  <r)  show  that  the  algorithm  then  effectively  expands  intensity  fluctua- 
tions . 

As  in  the  case  of  automatic  Haar  brightness  control,  individual  contrast  adjustment 
for  each  16-by-16  pel  block  will  tend  tj  create  artifacts.  This  can  be  avoided  by 
using  the  rms  value  of  <r  (computed  over  the  entire  frame)  in  the  proposed  algorithm. 


5.3  DYNAMIC  RANGE  COMPRESSION 

5.3.1  Problem  Definition 

g 

On  a somewhat  different  tack,  dynamic  range  compression  addresses  the  problem  of 
inefficient  use  of  the  available  display  gray  levels.  It  stretches  each  snail 
amplitude  interval  of  the  waveform  in  proportion  to  the  occupancy  of  that  interval. 
A long  interval  of  small  waveform  variations  about  a given  mean  value  will  be  accen- 
tuated. Large  spikes,  on  the  other  hand,  contribute  little  to  the  occupancy  of 
any  level  and  will  tend  to  shrink. 

True  dynamic  range  compression,  however,  is  a nonlinear  process.  Taking  g(x)  and 
h(x)  as  the  initial  and  final  waveforms, 


Required  Small-Signal  Gain  - ^ 


varies  as  a function  of  g rather  than  x and  cannot  be  readily  implemented  in  terms 
of  Haar  transforms.9  A further  goal  of  adaptive  signal  processing,  consequently, 
consists  in  replacing  this  process  by  an  approximation  suited  to  the  Haar  domain. 

g 

This  process  has  also  been  designated  Histogram  Equalization  in  the  literature. 
Consider  x^  • roots  of  g(x)  * gQ  * some  specified  amplitude.  Then, 

(dh/dg)g  ~ £ (dg/dx)  * ■ function  of  g . 

° all  x Xn 

n 


r 

■ 


5.3.2 


ynamic  Range  Compression  Solution 


Haar  implementation  requires  that  the  original  (nonlinear)  description  of  dynamic 
range  compression,  "A  small-signal  gain  assigned  to  every  amplitude  which  varies 
in  proportion  to  the  occupancy  of  that  level,"  be  replaced  with,  "A  gain  value 
assigned  to  each  sequency  and  proportional  to  the  occupancy  of  that  sequency." 


Sequency  occupancy  is  defined  as  ^ a 

m-1  mr 


In  the  one-dimensional  case,  both  descriptions  generally  promote  the  amplification 
of  long  intervals  of  low-amplitude  fluctuations  about  some  constantmean.  Both 
also  tend  to  deemphasize  sharp  spikes. 


Figure  5-13  illustrates  a case  in  point.  Original  waveform  g(x)  occupies  various 
amplitude  levels  according  to  the  numbers  to  the  left  of  the  g-axis.  In  the  usual 
nonlinear  approach  to  dynamic  range  compression,  these  numbers  specify  slopes  (small- 
signal  gains)  required  of  the  transducer,  as  shown.  Transducer  output  appears  as 
h(x) . 


The  linear  equivalent  output  meanwhile  appears  as  j(x).  Construction  of  this  wave- 
form involves  derivation  of  the  Haar  components  of  g(x)  as  listed  in  Table  5-2. 
Multiple-entry  sequencies  arrange  terms  in  order  of  increasing  offset.  Each  compo- 
nent is  then  amplified  by  the  gain  associated  with  its  sequency,  also  as  in  Table 


These  gains  are  obtained  by  multiplying  all  terms  of  given  sequency  r by  2^r^^  , 

12 

and  adding  the  absolute  values  of  the  resulting  numbers.  The  result  is  then  proc- 
essed through  an  inverse  Haar  transform  to  yield  j(x)  as  shown  in  Figure  5-13. 


Other  candidate  definitions  include  Y'  a , Y'  a etc.,  but  these  tend  to  mini- 

. mr  inr 

m*l  m*l 

mize  the  roles  played  by  the  smaller  a's  too  severely. 

* *An  exception  is  the  (-1)  or  dc  Haar  term.  This  term  does  not  alter  waveform  shape, 
but  only  regulates  its  average  value  (brightness).  Numerical  calculations  simplify 
when  this  particular  gain  is  chosen  equal  to  unity. 

12 

Such  a procedure  actually  yields  the  numbers  31,  21.21,  30,  and  42.44  for  sequen- 
cies 0,  1,  2,  and  3,  respectively.  For  simplicity,  the  numbers  have  been  divided 
by  10  and  rounded  to  the  nesreat  integer  to  yield  the  values  in  Table  5-2. 
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Figure  5-13.  DYNAMIC  RANGE  COMPRESSION  EXAMPLE 


Table  5-2.  RATIONALIZED  HAAR  COMPONENTS  OF  g(x) 


Average  values  of  the  ripple-groups  are  no  longer  constants,  but  seem  to  wander 

somewhat.  This,  however,  does  not  seem  particularly  objectionable.  Note,  toe, 

that  the  jump  between  the  ripple  groups  has  dropped  from  four  times  the  ripple  ampl 

tude  to  only  three  times.  Likewise,  the  single  spike  has  shrunk  from  eight  times 

the  amplitude  of  the  surrounding  ripples  to  something  less  than  six  times.  Finally 

the  average  ripple  amplitude  has  increased  from  (0.08)  x (g  = g • ) to  (0.11)  x 

max  min 

(h  - h . ).  All  three  changes  represent  dynamic  range  compression. 

ma  x min 

Graph  j(x)  has  not  compressed  the  waveform  as  much  as  h(x)  and  does  not.  therefore, 
use  the  gray  scale  quite  as  efficiently.  Note,  however,  that  h(x)  barely  yields 
a discernible  spike.  This  suggests  that,  beyond  a certain  point,  compression  may 
degrade  intelligibility.  The  j(x)  (Figure  5-13)  could  nevertheless  benefit  from 
further  compression;  various  modifications  of  the  linear  compression  algorithm  are 
possible . 


5.4  CONCLUSIONS 


Haar  transforms  can  serve  as  an  effective  tool  in  all  aspects  of  adaptive  signal 
processing.  Under  the  subheading  of  dc  restoration,  the  best  procedure  seemed  to 
be  the  identification  of  four  main  problem  areas  and  a separate  corrective  Haar 
algorithm  for  each  one.  In  applying  these  algorithms,  the  feasibility  of  convert- 
ing a nonlinear  aspect  of  dc  restoration  in  an  equivalent  linear  one  is  also  con- 
firmed . 
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Dynamic  range  compression  represents  a still  more  complex  problem  involving  non- 
linear  operations  in  the  image  domain.  Here,  equivalent  Haar  domain  operations 
had  to  proceed  via  an  analogy  between  gray-level  occupancy  and  sequency  occupancy. 
Results  show  obvious  imperfections  but  also  suggest  means  for  further  improvement. 
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SECTION  6 

BANDWIDTH  COMPRESSION 


6.1  INTRODUCTION 

In  sampling  TV  rate  imaging  sensors  (512  lines,  30  frames  per  second)  at  the  Nyquist 
rate  (8  megasamples/sec),  using  eight  bits  to  encode  the  image  intensity  (256  shades 
of  gray),  a data  rate  cf  64  megabits/sec  is  required.  Although  difficult  to  achieve, 
processing  at  this  high  data  rate  in  real  time  is  now  practical.  This  has  been 
demonstrated  by  the  operations  performed  using  the  Northrop  2-D  Haar  processor  in 
fulfilling  this  study  contract.  Unfortunately,  magnetic  recording  of  digital  images 
at  this  data  rate  is  not  practical  at  this  time.  Since  data  recording  has  many 
advantages  over  analog  video  recording,  it  would  be  highly  desirable  to  reduce  the 
data  bandwidth  by  such  an  order  of  magnitude  that  small,  low-cost  recording  devices 
could  be  used  in  conjunction  with  the  airborne  sensor.  This  would  be  particularly 
significant  in  a "pop-up"  mode  where  a vulnerable  vehicle  such  as  a helicopter  can 
pop-up,  snatch  several  frames  of  video  and  rapidly  remask  while  the  data  is  processed 
and  analyzed.  A reduction  in  data  bandwidth  also  greatly  reduces  the  difficulty 
of  transmitting  the  data  to  a remote  location  by  means  of  a radio  link  if  this  be- 
comes part  of  an  operational  requirement. 

This  study  is  an  evaluation  of  several  techniques  that  reduce  data  rates  by  removing 
redundancy,  compar'ng  them  with  simpler  filtering  techniques  that  reduce  resolution. 
By  using  the  Haar  transform  and  a combination  of  "thresholding"  and  the  "N  coeffici- 
ent" algorithm,  a data  rate  of  one  bit  per  pel  (bandwidth  reduction  of  8 to  1)  is 
easily  attainable.  This  does  not  take  into  consideration  other  techniques , such  as 
frame  rate  reduction,  change  detection  recording,  companding,  and  source  encoding 
schemes,  which,  when  incorporated  into  the  system,  allow  for  even  greater  compres- 
sion ratios. 

6.2  GENERAL  EXPERIMENTAL  PROCEDURE 

The  experimental  test  configuration  is  shown  in  Figure  6-1.  The  video  received 
from  the  TV  camera  is  sampled  at  8.19  MHz  and  then  digitized  to  eight  bits  (256 
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EXPERIMENTAL  TEST  CONFIGURATION 


g.  av  shades)  on  a linear  scale.  This  digital  image  is  used  as  the  non-compressed 
image  and  will  be  referred  to  as  the  "full-resolution"  image.  It  exhibits  an  in- 
formation content  of  8 bits/pel.  The  data  rates  determined  as  a result  of  the 
application  of  experimental  compression  algorithms  are  referenced  to  the  full- 
resolution  image.  For  example,  after  compression  the  total  number  of  bits  required 
to  represent  the  image  is  divided  by  the  number  of  pels  in  the  full-resolution 
image.  The  full-resolution  image  is  subdivided  into  16x16  pel  sectors  in  the  trans- 
form processor.  The  Haar  transform  of  each  section  is  then  computed  and  transmitted 
to  the  inverse  processor  where  the  digital  image  is  reconstructed.  The  transform 
format  of  the  processor  is  illustrated  in  Figure  6-2.  Image  data  compression  al- 
gorithms may  be  applied  either  to  the  Haar  transform  of  the  image  sector  or,  in 
some  cases,  to  the  image  section  directly. 

Two  images,  an  Electronic  Industries  Association  (EIA)  resolution  chart,  and  a FLIR 
thermogram,  were  used  as  test  imagery  for  the  initial  experiments.  The  resolution 
chart  provides  spatial  resolution  information  while  the  FLIR  image  provides  a more 
realistic  gray-scale  distribution. 

6.3  REDUCED  RESOLUTION  COMPRESSION 

The  information  content  of  a sampled  digital  image  may  be  compressed  directly  by 
reducing  either  the  number  of  image  samples  or  the  number  of  shades  of  gray  used 
to  represent  each  sample.  This  approach  to  bandwidth  reduction  is  referred  to  as 
reduced  resolution  compression.  Reduced  resolution  compression  is  the  simplest 
form  of  bandwidth  reduction  and  provides  a baseline  for  the  evaluation  of  more 
involved  image-compression  schemes. 

6.3.1  Scan  Line  Sampling 

In  a scanned  imaging  system,  such  as  TV  or  a parallel-scan  FLIR,  the  number  of  image 
samples  may  be  reduced  on  a line-by-line  basis.  This  approach  sacrifices  spatial 
resolution  in  only  the  scan  direction.  The  results  of  one  dimensional  sample  re- 
duction are  shown  in  Figure  6-3,  where  the  images  shown  were  made  by  reducing  the 
clock  frequency  to  the  output  register  of  the  inverse  Haar  processor.  As  is  shown 
in  the  figure,  the  horizontal  resolution  of  the  system  decays  fairly  rapidly  start- 
ing with  the  four  bit/pel  compression  rate. 
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6-2 . TRANSFORM  FORMAT 


6.3.2  Area  Sampling 


Spatial  resolution  may  also  be  reduced  in  two  dimensions  or  on  an  area  basis.  Fig- 
ure 6-4  shows  the  results  of  averaging  square  areas  of  pels  together  to  form  the 
output  image.  As  is  shown  in  Figure  6-4,  two  bits/pel  is  about  the  limit  of  ac- 
ceptable degradation  that  can  be  tolerated  using  this  approach.  Note  the  the  com- 
pression rate  achieved  by  area  sampling  is  the  square  of  the  rate  obtained  by 
reduced  line  sampling. 

6.3.3  Intensity  Sampling 

The  information  content  of  image  may  be  reduced  simply  by  using  fewer  bits  to  repre- 
sent the  intensity  of  each  image  sample.  The  effects  of  various  levels  of  quanti- 
zation of  intensity  are  illustrated  in  Figure  6-5.  A FLIR  image  was  used  here 
because  it  displays  more  shades  of  gray  than  the  video  test  pattern.  Note  that 
reducing  the  intensity  quantization  from  eight  bits/pel  (256  gray  shades)  to  six 
bits/pel  (64  gray  shades)  produces  only  a small  amount  of  degradation  in  the  image. 
At  four  bits/pel,  however,  false  intensity  contours  become  evident  in  the  back- 
ground in  the  vicinity  of  the  targets,  while  at  two  bits/pel,  much  scene  information 
is  lost.  The  number  of  bits  required  to  represent  the  image  intensity  will  also 
vary  with  the  type  of  imager,  future  processing  requirements  for  the  image,  and 
the  type  of  display  used.  FLIR's,  for  example,  exhibit  a very  wide  dynamic  range, 
thereby  possibly  requiring  more  than  eight  bits  of  intensity  information  for  ade- 
quate resolution.  Also,  if  the  image  information  is  to  be  processed  in  any  fashion 
(e.g.,  targets  detected)  a relatively  high  number  of  gray  shades  would  be  required. 

6.3.4  Reduced  Resolution  Summary 

The  results  of  reduced  resolution  compression  indicate  that  a full  resolution  image 
may  be  compressed  up  to  two  bits/pel  with  a small  degradation  in  quality.  The  goal 
of  subsequent  image  compression  effort  then  is  to  exceed  this  bandwidth  reduction 
with  little  additional  quality  losses. 


6.4  HAAR  TRANSFORM  COMPRESSION 


Rather  than  attempt  to  compress  the  image  directly,  a more  sophisticated  approach 
to  compression  is  to  transform  the  image,  using  an  orthogonal  transform,  and  then 
compress  the  result.  In  order  to  be  useful  for  compression,  the  transform  used 
in  this  approach  should  isolate  the  relevent  image  properties  from  the  pel-to-pel 
redundancies  most  images  exhibit.  Also,  the  computational  complexity  of  tie  trans- 
form must  be  compatible  with  the  data  rate  of  the  system.  The  two  dimensional  Haar 
transform  satisfies  both  of  these  requirements. 

The  Haar  transform  of  an  image  may  be  compressed  in  two  basic  ways.  The  first 
approach  is  to  reduce  the  number  of  bits  used  to  represent  each  Haar  coefficient 
The  second  method  of  compressing  is  to  reduce  the  number  of  Haar  coefficients  used 
to  reconstruct  the  original  image.  Both  approaches  to  compression  are  explored 
here . 

6.4.1  Sequency  Domain  Bit  Assignments 

The  2-D  sensor  processor  divides  the  Haar  transform  sequency  domain  into  15  regions 
or  sequency  groups.  A sample  amplitude  distribution  table  for  each  group  is  shown 
in  Figure  6-6.  Examination  of  the  amplitude  distributions  shows  that  in  most 
groups,  fewer  than  the  full-precision  nine  bits  (eight  bits  plus  sign)  are  required 
to  represent  the  elements  of  the  group.  By  bit  limiting  the  coefficient  amplitudes 
in  each  group,  about  a two-to-one  reduction  in  bandwidth  may  be  achieved  with  no 
visible  degradation  in  image  quality.  Figures  6-7  and  6-8  both  s\iow  an  original 
eight-bit /pel  image  along  with  an  image  compressed  by  assigning  fewer  bits  to  each 
sequency  group.  A table  showing  the  bit  assignments  used  is  also  given.  The  data 
rate  realized  by  this  approach  is  4.14  bits/pel. 

6.4.2  Coefficient  Elimination 

As  a result  of  the  information  compaction  properties  of  the  Haar  transform,  many 
of  the  transform  coefficients  contribute  little  or  no  information  about  the  image. 
As  a consequence,  these  terms  need  not  be  used  to  reconstruct  the  original.  Thus, 
the  image  information  may  be  compressed  by  eliminating  some  coefficients  from  con- 
sideration. Several  simple  forms  of  coefficient  elimination  are  considered  here. 
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6. 4. 2.1  Zonal  Filtering  - Zonal  filtering  is  performed  simply  by  zeroing  entire 
sequency  groups  or  zones  in  the  Haar  plane.  The  sequency  groups  that  are  retained 
(or  passed  through  the  zonal  filter)  are  those  that  are  deemed  a priori  to  contain 
the  important  scene  information. 

Several  zonal  filtered  images  are  shown  in  Figures  6-9  and  6-10.  In  each  case, 
the  filters  used  are  essentially  low-pass  filters.  However,  by  selecting  sequency 
groups  in  a judicious  manner,  a majority  of  the  edges  within  the  scene  may  still 
be  retained.  The  specific  edges  retained  are  also  determined  by  the  sequency  group 
selection.  This  edge- i sol  at ing  property  of  the  Haar  transform  can  be  seen  by  com- 
paring Figures  6-9c  and  6-9d.  While  both  images  are  compressed  to  the  same  level, 
the  diagonal  edges  of  Figure  6-9d  are  more  accurately  represented  than  those  of 
Figure  6-9c.  This  affinity  for  diagonal  edges,  however,  is  obtained  by  reducing 
the  resolution  of  horizontal  and  vertical  edges. 

6. 4. 2. 2 Thresholding  - Thresholding  eliminates  transform  coefficients  by  re- 
quiring them  to  exhibit  an  amplitude  greater  than  some  specific  threshold  value 

in  order  to  be  included  in  the  image  reconstruction.  This  approach  to  coefficient 
elimination  is  more  scene  adaptive  than  zonal  filtering  in  that  relatively  high 
amplitude  coefficients  may  be  passed  regardless  of  sequency.  However,  since  the 
position  of  the  above-threshold  coefficients  is  not  known  a priori  as  it  was  in 
zonal  filtering,  some  additional  positional  information  must  be  included  with  each 
coefficient  retained. 

The  number  of  bits  required  to  provide  coefficient  position  information  can  vary 
depending  on  the  coding  scheme  used.  For  example,  if  the  row  and  column  address 
is  used  to  locate  each  nonzero  coefficient,  a total  of  eight  bits  per  coefficient 
is  required.  This  requirement  may  be  reduced  by  scanning  the  Haar  plane  in  a 
specified  serial  fashion  and  encoding  the  number  of  zeroes  between  coefficients. 
This  approach  is  known  as  run-length  encoding.  A bit-efficient  comma-free  Huffman 
code  can  then  be  derived  from  the  statistics  of  the  run  lengths.  By  using  this 
type  of  encoding  while  scanning  the  Haar  plane  in  order  of  increasing  sequency, 
only  five  bits  of  position  information  (worst  case)  are  required  per  coefficient.* 


Powers,  J.  W.,  Nort.  72-331,  Image  Data  Compression  Using  Digital  Holography  (U) 
Powers  et  al , Nort.  73-293,  Imagery/Bandwidth  Compression  RPV  Study  (S) 
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(el  GROUPS  1-8.  11;  1.0  BIT/PEL  (f)  GROUPS  1-5,7,11;  QJ  BIT/PEL 

Figure  6-9.  ZONAL  FILTERED  TEST  PATTERN 


(b)  GROUPS  1-13,  ZA  BIT/PEL 


(a)  ALL  GROUPS;  8 BIT/PEL 


(d)  GROUPS  1-10.  12.  1.6  BIT/PEL 


(e)  GROUPS  1-9,  11,  12.  1.6  BIT/PEL 


<«)  GROUPS  1-5,  7,  11.  0.8  BIT/PEL 


la)  GROUPS  1-8.  It;  1.0  BIT/PEL 
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Once  the  positional  information  requirements  are  set,  the  compression  rates  for 
different  levels  of  thresholding  may  be  determined.  Figures  6-11  and  6-12  show 
the  effects  of  various  threshold  levels  in  the  sequency  domain.  The  corresponding 
data  rates  shown  in  the  figures  were  determined  by  multiplying  the  number  of  above- 
threshold coefficients  in  each  group  by  the  group's  bit  assignment  (Figure  6-7) 
plus  five  bits  of  position  information.  The  resultant  values  are  then  rummed  over 
the  entire  image  and  scaled  by  the  number  of  pels  to  arrive  at  the  number  of  bits 
per  pel.  Computing  the  data  rate  in  this  fashion,  however,  assumes  the  use  of 
a large  data  buffer  to  accommodate  the  variations  in  frequency  of  above-threshold 
coefficients. 

While  the  compression  ratios  obtainable  using  thresholding  are  much  greater  than 
those  observed  in  zonal  filtering,  the  storage  requirements  make  the  practical 
use  of  thresholding  unattractive.  A compromise  approach  that  exhibits  much  of  the 
scene  adaptability  of  thresholding  while  providing  the  advantages  of  a relatively 
constant  data  rate  will  be  examined  next. 

6. 4. 2. 3 N Coefficient  Selection  - A method  of  assuring  a constant  data  rate 
with  minimal  memory  requirements  is  to  require  that  for  each  image  sector,  exactly 
N coefficients  are  retained.  The  N coefficients  are  selected  by  setting  a threshold 
as  before  and,  while  scanning  the  Haar  domain  in  order  of  increasing  sequency, 
retaining  the  first  N above-threshold  terms.  Thus,  if  there  are  more  than  N sig- 
nificant coefficients  in  an  image  sector's  Haar  transform,  the  ones  relating  the 
higher  sequency  information  will  be  excluded  from  the  reconstruction  process. 
Conversely,  if  fewer  than  N above-threshold  coefficients  are  present  in  a given 
sector,  an  appropriate  number  of  zeroes  may  be  included.  This  compression  approach 
requires  storage  of  only  N elements  rather  than  a majority  of  the  compressed  image 
f rame . 

Figures  6-13  and  6-14  show  the  effects  of  this  type  of  compression  on  the  two  test 
images.  Note  that  for  fairly  similar  image  quality,  the  compression  rates  obtained 
are  comparable  to  those  observed  by  thresholding  only.  Also,  comparison  of  Figures 
6-13c  and  6-14c  shows  that  N coefficient-selection  provides  an  image  with  better 
resolution  than  an  under-sampled  one  while  requiring  only  one-half  as  m any  bits  per  pel. 
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la)  ORIGINAL 


(b)  1.87  BIT/PEL;  THRESHOLD  AT  2 


(»)  0 62  BIT/PEL;  0*  27.1,2.1.2.4.2.2.2.2.3.3.3.3.3 

Figure  6-12.  EFFECTS  OF  THRESHOLD  LEVELS  IN  SEQDENCY  DOMAIN,  FUR  NO.  1 
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(c)  a 24  BIT/PEL;  THRESHOLD  AT  3 


(d)  1.78  BIT/PEL.  0-  1.1.2.U;  2.2.1. 1.1;  2.2.2.2,3 


(b)  1.47  BIT/PEL;  N = 51 


U)  ORIGINAL 


Id)  0.83  BIT/PEL.  N = 22 


|c)  0.99  BIT/PEL;  N = 31 


(•I  0.55  BIT/PEL;  N * 15 


f>-13.  THRESHOLD  AND  N-COEFFICIENTS  TEST  PATTERN 
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The  performance,  simplicity  and  system  requirements  of  the  N coefficient  selection 
algorithm  make  it  the  optimal  compression  algorithm  within  the  sensor  processor's 
immediate  capability.  Five  FLIR  images  that  have  been  compressed  to  various  levels 
using  this  algorithm  are  shown  in  Figures  6-15  through  6-19.  The  figures  show 
that  a data  rate  of  1.15  bits/pel  is  easily  obtainable  for  each  image. 

6.5  IMAGE  DESCRIPTORS  FOR  HAAR-COMPRESSED  IMAGES 

The  set  of  image  descriptors  was  calculated  for  each  of  the  processed  images  included 
in  this  section  (see  paragraph  7.2  for  a derivation  of  the  "image  descriptor"). 

Table  6-1  lists  the  average  tone,  texture,  and  edge  values  computed  for  the  entire 
image.  In  each  case,  the  tone  value  remains  approximately  the  same  throughout 
processing.  It  is  included  as  a check  for  each  image/computer  set-up.  Since  targets 
are  separated  from  clutter  primarily  by  their  edge  content,  successful  bandwidth 
compression  would  try  to  preserve  edges  at  the  expense  of  texture.  Therefore, 
in  a perfect  bandwidth  reduction  scheme,  the  ratio  of  the  average  edge  to  the 
average  texture  divided  by  the  bandwidth  reduction  would  remain  constant.  (This 
presupposes  ideal  edge  and  texture  descriptors.)  As  can  be  seen  from  the  table, 
the  edge  descriptors  show  a minimum  amount  of  decrease,  while  the  texture  falls 
off  rapidly  with  bandwidth  reduction. 

A figure  of  merit  (f)  has  been  devised  to  offer  a rough  grading  of  the  effects 
of  compression  on  the  different  image  samples: 


(e/x)  (b/8) 

e /x 
o o 


where  b is  bits/pel  and  e^  and  x are  edge  and  texture  average  descriptors  of 
original . 


As  is  noted  in  the  table,  f decreases  as  the  compression  increases  in  almost  all 
cases.  Looking  at  the  images,  one  can  see  deterioration  increasing  with  compres- 
sion, which  tends  to  validate  the  values  of  f.  This  relationship  holds  for  the 
test  pattern  and  each  of  the  FLIR  images.  If  0.5  were  arbitrarily  selected  for 
a cut-off  value,  one  bit  per  pel  would  be  obtainable,  using  the  N-coe f f ic ient 
algorithm  with  threshold. 


Ic)  0.99  BIT/PEL 


(d)  0.75  BIT/PEL 


Figure  6-16.  THRESHOLD  AND  N-COEFF  IC  I F.NTS  ( NVL  NO.  6) 
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Id)  a 75  BIT/PEL 


U)  ORIGINAL 


lb)  1.47  BIT/PEL 


U)  ORIGINAL 


(bl  1 47  BIT/PEL 


(•I  0.52  BIT/PEL 

Figure  6-18.  THRESHOLD  AND  N COEFFICIENTS  (NVL  NO.  8) 


6-25 


Table  6-1.  DESCRIPTORS  FOR  HAAR- COMPRESSED  IMAGES 


Figure 

Image  - Process 

6-7 

Test  Pattern 

a 

8-bit/pel  (original) 

b 

4 . 14-bit/pel 

6-8 

FLIR  #1 

a 

8-bit /pel (or iginal ) 

b 

4. 14-bit/pel 

6-9 

Test  Pattern 

a 

Original 

b 

Zonal  2.4-bit/pel 

c 

Zonal  1.6-bit/pel 

d 

Zonal  1.6-bit/pel 

e 

Zona  1 1 . 0-bit/ pe 1 

f 

Zonal  0.8-bit/pel 

6-10 

FLIR  #1 

a 

Original 

b 

Zonal  2.4-bit/pel 

c 

Zonal  1.6-bit/pel 

d 

Zonal  1.6-bit/pel 

e 

Zonal  1 .0-bit/pel 

f 

Zonal  0.8-bit/pel 

6-11 

Test  Pattern 

a 

Or iginal 

b 

Threshold  1.87-bit/pel 

d 

Threshold  1.78-bit/pel 

e 

Threshold  0.83-bit/pel 

c 

Threshold  0.55-bit/pel 

6-12 

FLIR  #1 

a 

Or igina l 

b 

Threshold  1.87-bit/pel 

d 

Threshold  1.78-bit/pel 

e 

Threshold  0.62-bit/pel 

c 

Threshold  0.24-bit/pel 

6-13 

Test  Pattern 

a 

Original 

b 

N-coe f f ic ient  N=  5 1 (1.47) 

c 

N-coef f icient  N= 3 1 (0.49) 

d 

N-coef f icient  N=22  (0.75) 

e 

N-coeff icient  N= 1 5 (0.52) 

♦Figure  of  merit 


Average 

Tone 

Text . 

Edge 

32 

568 

1747 

32 

524 

1239 

22 

416 

516 

22 

399 

501 

32 

602 

1677 

32 

214 

1462 

32 

213 

1398 

32 

159 

1371 

32 

177 

1364 

32 

158 

1336 

22 

483 

582 

22 

206 

472 

22 

169 

438 

22 

130 

430 

22 

178 

414 

22 

121 

391 

32 

637 

1612 

32 

339 

1615 

32 

287 

1624 

34 

186 

1633 

34 

120 

1665 

22 

454 

546 

22 

164 

673 

22 

144 

648 

22 

75 

607 

22 

50 

595 

32 

731 

1712 

31 

221 

1481 

32 

142 

1388 

32 

103 

1332 

32 

58 

1236 

i-2 


Table  6-1.  DESCRIPTORS  FOR  HAAR- COMPRESSED  IMAGES  (Continued) 


Figure 

Image  - Process 

Average 

f* 

Tone 

Text . 

Edge 

6-14 

FLIR  #1 

■n 

a 

Original 

22 

449 

546 

b 

N-coef f ic ient  N=  5 1 (1.47) 

23 

103 

505 

ESS 

c 

N-coef f icient  N=  3 1 (0.99) 

23 

80 

422 

EO 

d 

N-coeff icient  N=22  (0.75) 

23 

62 

388 

1K9 

e 

N-coe f f ic ient  N=  1 5 (0.52) 

23 

41 

355 

0.47 

6-15 

FLIR  #4 

a 

Or iginal 

18 

286 

723 

1.00 

b 

Threshold  & N-coef f ic ient  1.47-bit/pel 

19 

87 

729 

0.62 

c 

Threshold  & N-coef f ic ient  0.99-bit/pel 

18 

69 

630 

0.45 

d 

Threshold  & N-coef f icient  0. 75-bit/pel 

18 

52 

565 

0.41 

e 

Threshold  & N-coef f ic ient  0.52-bit/pel 

18 

42 

524 

0.32 

6-16 

FLIR  #6 

a 

Or  igina 1 

23 

567 

1.00 

b 

Threshold  & N-coef f ic ient  1.47-bit/pel 

24 

533 

0.90 

c 

Threshold  & N-coef f ic ient  0.99-bit/pel 

24 

437 

0.68 

d 

Threshold  6 N-coef f ic ient  0.75-bit/pel 

23 

390 

0.57 

e 

Threshold  & N-coef f icient  0.52-bit/pel 

23 

367 

0.48 

6-17 

FLIR  #7 

a 

Original 

30 

1029 

1 .00 

b 

Threshold  6 N-coef ficient  1.47-bit/pel 

31 

200 

959 

0.55 

c 

Threshold  & N-coef ficient  0.99-bit/pel 

30 

163 

861 

0.41 

d 

Threshold  & N-coef f ic ient  0.75-bit/pel 

30 

116 

813 

0.41 

e 

Threshold  & N-coef f ic ient  0.52-bit/pel 

30 

87 

776 

0.36 

6-18 

FLIR  #8 

a 

Original 

16 

353 

573 

1.00 

b 

Threshold  & N-coef f ic ient  1.47-bit/pel 

15 

94 

536 

0.65 

C 

Threshold  6 N-coef f ic ient  0.99-bit/pel 

15 

83 

475 

0.44 

d 

Threshold  & N-coeff icient  0.75-bit/pel 

15 

62 

431 

0.41 

Threshold  & N-coef f ic ient  0.52-bit/pel 

15 

45 

407 

0.37 

FLIR  #10 

Original 

18 

299 

444 

1 .00 

reshold  & N-coef f ic ient  1.47-bit/pel 

18 

61 

416 

0.84 

1 ■ -h.ld  & N-coe f f ic ient  0.99-bit/pel 

18 

51 

0.43 

& N-coef f ic ient  0.75-bit/pel 

18 

38 

308 

0.51 

S N-i oef f ic ient  0.52-bit/pel 

18 

28 

279 

0.43 

6.6  SUMMARY  AND  CONCLUSIONS 


The  Haar  transform  provides  a practical  method  for  removing  redundancy  from  FLU 
images  in  real  time.  Four  techniques  for  reducing  data  in  the  Haar  domain  have 
evaluated.  A judicious  assignment  of  coding  bits  for  Haar  sequency  groups  more 
than  halves  the  bandwidth  required  to  represent  the  transformed  image.  Zonal 
filtering  provides  a simple  method  for  further  reducing  bandwidth  by  omitting  ei 
groups  of  coefficients,  but  zonal  filtering  is  scene-dependent  and  tends  to  fav< 
either  edges  or  diagonals.  Thresholding  is  a more  efficienc  means  of  coefficiei 
reduction.  It  is  highly  adaptive  to  different  scenes;  unfortunately,  it  require 
variable  length  memory  buffer  and  a complex  encoder.  A good  compromise  is  the  I 
coefficient  selection  algorithm,  which  permits  thresholding  with  a fixed-size  bi 
and  a simple  encoder.  As  a result  of  the  experiments  performed,  Northrop  recom- 
mends a combination  of  reducing  coding  bits  assigned  to  Haar  sequency  groupings 
limited  zonal  filtering  and  the  N-coef f icient  selection  algorithm.  When  used  w 
other  reduction  methods,  i.e.,  frame  rate  reduction,  this  should  provide  a band1 
reduction  of  25  to  1 or  greater  for  representative  FLIR  images  without  losing  si 
targets  in  the  background  clutter. 


7.1  INTRODUCTION 


It  is  generally  accepted  that  evaluation  of  processed  imagery  is  a difficult  problem 
Mean  square  error  has  been  used  as  a fidelity  criterion  for  most  image  coding  tech- 
niques and  is  believed  not  to  compare  well  with  human  evalution.  Mean  square  error 
does  not  make  sense  in  the  areas  of  image  enhancement,  for  example.  Also,  in  most 
cases  of  image  restoration,  it  cannot  be  obtained  since  no  reference  image  is  avail- 
able. Quantif ication  of  the  effect  of  various  image  enhancment  and  restoration 
techniques  requires  a measure  that  can  easily  be  obtained  from  the  image  and  agrees 
well  with  the  human  subjective  evaluation. 

The  development  of  a set  of  Image  Descriptors  (ID's)  that  are  based  on  the  statis- 
tics of  an  image's  two-dimensional  rationalized  Haar  transform  is  presented  (see 
Section  1).  The  ID's  define  a division  of  pictorial  information  into  two  major 
categories:  luminance  (dc  information)  and  activity  (ac  information) . Luminance 

is  further  subdivided  into  brightness,  given  by  the  average  gray  level,  and  tone, 
which  is  determined  from  the  variance  of  a local  ensemble  average  gray  level.  A 
division  of  activity  into  texture  and  edge  information  is  accomplished  by  examining 
an  edge  criterion  function  for  each  ac  term  of  the  image's  Haar  transform. 


The  ID's  can  be  used  to  quantify  the  characteristics  of  images  both  before  and  after 
the  application  of  various  image  processing  algorithms.  The  resultant  changes  in 
the  ID's  can  then  be  used  to  detect  the  net  effect  of  processing  on  the  image. 

When  each  ID  is  viewed  as  an  element  of  a descriptor  vector,  an  image  description 
space  is  then  defined.  The  evaluation  of  images  within  this  description  space  was 
a part  of  this  study. 

The  effects  on  the  ID's  observed,  due  to  the  application  of  various  algorithms, 
are  reviewed  in  this  section.  Some  conclusions  and  recommendations  based  on  the 
descriptors'  performance  are  also  discussed. 
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7.2  IMAGE  DESCRIPTION  IN  THE  HAAR  DOMAIN 


l 

[■ 


To  facilitate  the  computation  of  the  image  descriptors,  their  definition  must  be 
compatible  with  the  data  handling  format  of  the  sensor  processor.  The  transform 
format  of  the  sensor  processor  is  such  that  the  Haar  transform  is  evaluated  for 
individual,  nonoverlapping,  16-by-16  pel  blocks.  Local  descriptors  are,  therefore, 
obtained  for  each  block  and  then  combined  to  form  global  descriptors  that  apply 
to  the  entire  image  or  areas  of  interest  within  the  image.  The  definitions  of  the 
local  descriptors  and  their  rules  of  composition  to  form  the  global-  or  image- 
descriptors  are  presented  here. 

7.2.1  Local  Image  Descriptors 

7. 2. 1.1  Local  Tone  - The  local  tone  of  the  it^1  block  of  the  digitized  image  is 
the  average  intensity  of  the  block,  and  is  represented  by  the  dc  term  in  the  Haar 
domain : 
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7. 2. 1.2  Local  Activity  - The  local  activity  of  the  i*"^1  block  is  defined  as  the 
ac  energy  of  the  block.  This  is  given  by  a weighted  sum  of  the  sum  of  the  squares 
of  the  ac  terms  of  its  rationalized  Haar  tranform: 
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The  coefficients  w are  weights  to  measure  the  importance  of  transform  coeffi- 

r,m,p,n 

cients  ar  m p n*  For  initial  analysis,  the  weights  were  taken  to  be  unity.  The 
quantity  c^  can  be  used  as  a measure  of  activity.  If  the  subpicture  has  no  detail, 
then  c^  = 0.  If  it  has  a lot  of  detail,  then  c^  is  large. 
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The  term  by  itself  is  insufficient  to  characterize  the  information  in  block  i. 
Two  blocks  may,  for  example,  have  comparable  activity  when  one  contains  significant 
edges  (outline  of  an  object,  etc.)  while  the  other  contains  only  high-power  white 
noise.  For  this  reason,  activity  is  further  subdivided  into  texture  activity  and 
edge  activity. 


The  separation  of  activity  into  texture  and  edge  takes  advantage  of  the  local  proper 
ties  of  the  Haar  basic  functions.  Within  a sequency  group,  the  Haar  functions  are 
simple  translations  of  one  another.  By  viewing  each  Haar  function  as  an  image 
primitive  and  the  corresponding  Haar  sequency  component  as  a measure  of  that  func- 
tion's contribution  to  the  image,  the  concept  of  texture  being  a repetition  of  local 
image  primitives  may  be  extended  to  the  Haar  domain.  When  the  total  power  of  a 
sequency  group  is  distributed  relatively  evenly  among  the  elements  of  the  group,  eac 
of  the  primitives  contribute  an  equal  amount  of  information  and  a texture  is  repre- 
sented. When  the  group  contains  a few  high-power  "spikes,"  they  are  deemed  to  be 
the  result  of  edges  in  the  image  domain.  For  example,  consider  the  upper  portion 
of  Group  li  in  Figure  1-5  (Section  1).  If  the  power  of  each  of  the  coefficients 
is  equal,  a texture  of  vertical  stripes  is  represented.  If  there  is  one  coefficient 
with  significantly  greater  power  than  its  neighbors,  then  an  edge  similar  to  the 
corresponding  Haar  function  is  present.  The  goal  of  the  edge-texture  separation 
procedure  is  to  essentially  sort  out  the  edge  terms  in  a group.  The  remaining  power 
in  the  group  is  then,  by  default,  due  to  texture. 

7 2.1.3  Local  Edge  Activity  - The  local  edge  activity  of  the  it^1  block  is  the 

ac  energy  that  satisfies  the  edge  criterion  function.  In  subjective  terms,  the 

edge  criterion  function  dictates  that  if  the  power  of  an  ac  coefficient  contributes 

more  than  its  "fair  share"  to  the  total  power  of  its  sequency  group,  then  that  term 

is  the  result  of  an  edge  in  the  image  domain.  More  specifically,  if  denotes 

the  set  of  edge  terms,  then  a is  an  element  of  di  when 
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where  o is  an  empirical  edge  power  th-eshold  for  the  rp  sequency  group  and  Srp 
is  the  relative  entropy  of  the  group.  If  P( ariBpn ) is  the  probability  that  a^^^will 
occur  within  the  group,  then 


P(a  ) log_P(a  ) 
mn  rmpn  2 rmpn 
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The  values  of  o^are  shown  in  Figure  7-1.  As  can  be  seen  in  the  figure,  only  15 

groups  are  used.  This  reflects  the  assumption  that  horizontal  and  vertical  edges 

are  equally  likely.  The  values  of  a were  chosen  empirically,  but  approximate 

^ ^ 

1/1+K  where  K is  the  number  of  Haar  functions  in  the  rp  group  to  which  a straight 
rp  rp 

edge  would  contribute  power.  This  approximate  relationship  for  shallows  diagonal 
edges  to  be  detected  more  readily. 

The  purpose  of  weightings  with  the  group's  relative  entropy  can  be  seen  by  exam- 
ining  the  two  representative  histograms  of  coefficients  of  the  rp  group  shown  in 
Figure  7-2. 

The  power  that  the  edge  terms  would  require  in  order  to  differ  significantly  from 
the  other  terms  is  greater  in  the  flat  distribution  of  Figure  7-2B  than  in  the  dis- 
tribution of  Figure  7-2A.  A good  measure  of  the  distribution  flatness  is  its  entropy. 
The  entropy  is,  therefore,  used  to  weight  the  edge-power  threshold. 


If  we  let  e^  be  the  local  edge  activity,  then 
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The  term  a . , . . can  be  excluded  from  equation  (7-5)  by  ensuring  that  it  will 
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never  satisfy  the  edge-criterion  function. 


Figure  7-1.  GROUP  EDGE  WEIGHTING,  »rp 
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figure  7-2.  REPRESENTATIVE  HISTOGRAMS  OF  COEFFICIENTS  OF  jth  GROUP 


7. 2. 1.5  Examples  of  Local  Descriptors  - The  two  sample  images  shown  in  Figure 
7-3,  "USC  Girl"  and  "Moon,"  form  a good  data  set  for  the  expository  development 
of  the  descriptors.  It  is  desirable  for  the  test  data  set  to  be  kept  as  small  as 
possible  for  demonstration  purposes  and  yet  be  sufficiently  diverse  to  exhibit  a 
reasonable  cross  section  of  image  characteristics.  It  is  felt  that  the  two  selected 
images,  Girl  and  Moon,  provide  enough  local  diversity  individually  and  global  di- 
versity between  each  other  to  provide  an  insight  into  the  behavior  of  the  descriptor 
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MOON 


Figure  7-3.  SAMPLE  IMAGES 


Each  image  consists  of  256  x 256  pels  that  are  subdivided  into  a 16  x 16  array  of 
16  x 16  pel  blocks.  The  local  descriptors  were  computed  for  each  block  of  both 
sample  images.  Representative  results  for  the  Girl  image  are  shown  in  Figures  7-4 
and  7-5. 

Examination  of  the  blocks  relating  the  local  tone  of  the  Girl  image  (Figure  7-4) 
shows  that  the  results  obtained  are  in  good  agreement  with  subjective  evaluation. 

This  is  to  be  expected  since  the  local  tone  is  nothing  more  than  the  average  gray 
level  of  the  16  x 16  pel  block.  Dark  areas  of  the  picture  have  a correspondingly 
lower  local  tone  than  bright  areas.  Subjective  agreement  is  also  good  for  local 
activity.  Busier  areas  of  the  picture  have  correspondingly  higher  values  of  activity 

Figure  7-5  provides  a good  example  of  the  division  of  edge  and  texture  activity. 

The  two  blocks  shown  exhibit  comparable  activity  and  yet  contain  significantly 
different  information.  The  block  containing  the  edge  of  the  Girl's  face  exhibits 
high  edge  activity  (edge : texture  ratio  is  3.5:1)  while  in  the  block  containing  the 
flower,  texture  activity  is  the  majority  contributor  ( edge : texture  ratio  is  1:1.3). 
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Figure  7-5.  EXAMPLES  OF  LOCAL  TEXTURE  AND  EDGE  ACTIVITY 


7.2.2  Global  Image  Descriptors 


Obtaining  global  descriptors  of  tone,  edge  activity,  and  texture  activity  requires 
a suitable  combination  of  the  quantities  computed  on  the  subpictures  of  the  image. 
Since  the  edge  and  texture  activity  add  up  to  local  activity  numerically,  only  the 
results  of  local  tone  and  activity  will  be  developed  herein.  The  rules  for  the  com- 
bination of  local  edge  and  texture  activity  will  be  equivalent  to  those  for  the 
combination  of  local  activity. 


The  local  tone  and  activity  have  been  computed  on  all  of  the  256  blocks  of  sample 
images  Girl  and  Moon.  A global  index  must  be  obtained  from  the  set  of  local  values. 
To  aid  in  the  comparison  of  the  images,  some  first-order  statistics  of  the  local 
descriptors  were  computed  as  shown  in  Table  7-1. 
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Table  7-1.  STATISTICS  OF  LOCAL  TONE  AND  ACTIVITY 


Girl 

Moon 

Local  Tone 

T 

Mean 

73 

127 

Minimum 

10 

30 

Maximum 

205 

178 

Standard  Deviation 

35 

22 

Local  Activity 

Mean 

2,367 

2,065 

Minimum 

1 30 

489 

Maximum 

23,606 

9,975 

Standard  Deviation 

j 

3,189 

1,391 

i 

7.2.2.  I Image  Tone  - As  can  bt-  seen  from  Table  7-1,  the  mean  of  the  local  tone 
(average  brightness)  is  greater  for  Moon  than  for  Girl.  Subjective  evaluation, 
however,  indicates  that  the  Girl  image  has  the  most  global  tone.  Image  tone  should 
relate  the  diversity  of  the  local  brightness  rather  than  the  brightness  itself. 

The  standard  deviation  of  the  local  tine  then  i .->  taken  as  a measure  of  the  global, 
er  image,  tone.  Let  the  image  tone  be  denoted  by  T;  then 


(t . - B)2 


N - 1 


1/2 
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where  t.  is  the  local  tone  of  the  i^1  block,  B is  the  average  brightness  of  the 
image,  and  N is  the  number  of  16  x 16  pel  blocks  in  the  image.  The  source  of  the 
difference  in  standard  deviations  of  the  local  tone  of  the  two  images  can  be  seen 
readily  by  examining  the  histograms  of  local  tone  shown  in  Figures  7-6  and  7-7. 

The  preceding  expression  for  image  tone  exhibits  the  following  characteristics: 
When  the  local  tone  of  each  image  block  is  approximately  equal,  i.e.,  the  total 
picture  is  of  uniform  average  intensity,  the  image  tone  is  near  zero.  When  the 
local  tone  changes  significantly  from  block  to  Slock,  the  image  tone  is  large. 
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7. 2. 2. 2 Image  Activity  - A comparison  of  the  statistics  of  the  local  activity 
of  the  Girl  and  Moon  images  (Table  7-1)  reveals  a significant  difference  in  their 
standard  deviations.  The  standard  deviation  of  the  local  activity  is  not,  however, 
taken  as  a measure  of  image  activity.  The  reason  for  this  decision  becomes  clearer 
when  we  consider  the  following  example:  In  a given  image  assume  that  each  16  x 
16  pel  block  contains  either  sufficient  edge  or  texture  information  to  give  it  a 
local  activity  of  cq,  a constant.  Each  block  may  be  completely  different  from  its 
neighbors;  the  only  similarity  is  their  local  activity.  If  the  standard  deviation 
of  the  local  activity  were  the  measure  for  image  activity,  the  image  activity  would 
be  zero.  But,  if  one-half  of  the  blocks  of  this  image  were  changed  to  a uniform 
gray,  i.e.,  made  to  contain  no  ac  energy,  the  standard  deviation  of  the  local  activity 
would  increase.  Such  a result  is  counterintuitive  to  a measure  of  image  activity 
The  standard  deviation  of  the  local  activity  does,  however,  impart  some  information 
about  the  total  image.  This  parameter  is  dubbed  "image  diversity"  but,  along  with 
image  brightness,  is  not  presently  considered  a necessary  element  of  the  global 
image  descriptors. 


The  measure  to  be  used  for  image  activity  is  the  mean  of  the  local  activity.  Image 
activity  thus  measured  would  rise  and  fall  in  agreement  with  subjective  evaluation 
in  the  example  given.  Examination  of  Table  7-1  shows  that  the  Girl  image  is  about 
15  percent  more  active  than  the  Moon  image.  Let  C denote  the  global,  or  image, 
activity, 
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where  c^  is  the  activity  of  the  i^  block.  Similar  expressions  appK 
image  texture  activity  and  E,  the  image  edge  activity;  i.e., 
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where  x.  and  e.  are 
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The  statistics  of  the  division  of  local  activity  into  edge  and  texture  activity 
for  the  two  sample  images  are  shown  in  Table  7-2. 


Table  7-2.  STATISTICS  OF  LOCAL  EDGE  AND  TEXTURE  ACTIVITY 


Girl 

Moon 

Local  Edge  Activity 

Mean 

1,191 

850 

Maximum 

15,147 

4,937 

Minimum 

99 

195 

Standard  Deviation 

1,754 

737 

Local  Texture  Activity 

Mean 

1,176 

1,214 

Maximum 

14,675 

5,783 

Minimum 

5 

223 

Standard  Deviation 

1,635 

776 

Table  7-2  shows  that  the  global  texture  activities  of  the  two  images  are  com- 
parable, but  the  Girl  image  displays  more  edge  activity.  This  is  in  good  agree- 
ment with  initial  subjective  analysis. 

7. 2. 2. 3 Global  Descriptor  Vector  The  three  global  features  used  as  image  de- 

sc  ptois  are  T,  image  tone;  E,  image  edge  activity;  and  X,  image  texture  activ- 
ity. These  three  descriptors  are  defined  to  be  the  components  of  the  image  descrip- 
tor vector,  n^=[T,E,X] . Thus,  the  descriptors  of  an  image  define  a point  in 

a three-dimensional  image  descriptor  space. 

7.3  DESCRIPTOR  RESULTS 

In  each  experiment  the  image  descriptors  have  been  used  to  evaluate  image  content 
both  before  and  after  processing.  Specifically,  the  descriptors  of  images  of  various 
states  of  focus,  edge  enhancement,  and  compression  have  been  computed.  A summary 
of  each  of  these  results  is  presented  below. 


7.3.1  Focus  Results 


During  the  autofocus  experiment,  the  behavior  of  texture  and  edge  activity  as  a 
function  of  lens  position  was  determined  for  the  video  test  pattern  as  well  as 
several  FLIR  images.  This  was  performed  in  a fashion  similar  to  the  other  focus 
measures,  i.e.,  by  evaluating  the  focus  measure  at  each  point  through  one  complete 
lens  cycle.  The  area  of  the  test  pattern  for  which  the  descriptors  were  computed 
was  more  inclusive  than  that  used  for  the  other  measures.  Figure  2-13  shows  the 
test  pattern  with  the  descriptor  input  areas  flagged  by  increased  brightness.  These 
areas  were  chosen  so  that  texture  and  edge  ould  be  computed  for  both  the  center 
of  the  image,  which  is  primarily  texture,  and  the  borders  of  the  image,  which  are 
composed  primarily  of  two  singular  edges.  On  the  FLIR  iamges,  both  target  and  back- 
ground areas  were  examined.  The  background  areas  are  identical  to  those  shown  in 
Figure  2-15.  For  target  input  information,  a 3 by  4 block  of  image  sectors  was 
centered  about  one  of  the  targets  in  each  image. 

The  curves  relating  the  behavior  of  texture  and  edge  activity  as  a function  of  lens 
position  are  shown  in  Figures  2-14  and  2-27  through  2-30.  It  can  be  seen  that  while 
texture  and  edge  activity  show  approximately  the  same  sensitivity  to  focus,  regard- 
less of  the  type  of  input  data,  the  texture  curve  is  much  noisier.  Averaging  the 
values  of  texture  activity  over  several  lens  cycles  would  probably  reduce  this 
noise  level  substantially,  but  the  increase  in  computation  time  for  such  an  approach 
would  be  prohibitive  in  a real-time  system.  For  the  background  type  data  seen  in 
the  sample  FLIR  images  provided  by  NVL,  the  division  of  activity  into  texture  and 
edge  information  provides  little  to  no  additional  information  about  the  image's 
state  of  focus. 

7.3.2  Enhancement  Results 


The  ID's  were  used  in  conjunction  with  subjective  analysis  to  evaluate  the  edge 
enhancement  algorithms  developed  in  Section  4.  The  ID's  for  the  enhanced  versions 
of  FLIR  Image  1 given  in  Table  2-6  show  that  edge-enhanced  images  possess  substan- 
tially larger  edge  activities  than  non-enhanced  imaged  with  only  a marginal  in- 
crease in  texture  activity.  However,  some  anomalies  do  exist  in  the  relative  levels 
of  edge  activity  between  subjectively  equivalent  images.  Figures  2-15a  and  2-15b 
for  example,  exhibit  edge-texture  ratios  of  about  10:1  and  2:1  respectively,  yet 
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appear  the  same  on  the  display.  This  inconsistency  in  edge  description  is  due  to 
the  uniform  weight  w given  each  Haar  term  in  equation  (7-2).  With  a uniform 
weight,  certain  sequency  groups  containing  mostly  noise  information  (e.g.,  Groups 
14  and  15)  may  have  their  contribution  to  the  total  image  description  overemphasized 
This  problem  was  circumvented  to  some  extent  by  not  subjecting  the  noise  groups 
to  the  enhancment  algorithms.  The  noise  contribution  from  these  groups  is  then 
equivalent  in  both  the  original  and  enhanced  versions  of  the  image.  Using  this 
approach,  an  edge-texture  ratio  of  about  2:1  was  regularly  preferred  by  the  evalu- 
ators. This  result  will  be  altered  slightly  when  alternate  sets  of  coefficient 
weights  are  developed  that  will  more  closely  match  the  contribution  of  each  sequency 
group  to  the  MTF  of  the  system.  This  is  a current  development  effort. 

7.3.3  Compression  Results 

The  behavior  of  the  image  descriptors  resulting  from  different  levels  of  bandwidth 
reduction  was  evaluated  as  part  of  the  compression  experiment.  The  basic  goal  of 
image  compression  is  to  suppress  redundant  texture  information  while  retaining 
pertinent  edge  features.  As  shown  in  Table  6-1,  increased  levels  of  compression 


reduce  an  image's  edge  content  slightly  while  the  texture  descriptor  falls  off 
rapidly. 


A figure  of  meter  (f)  was  devised  to  offer  a rough  grading  of  the  effects  of  com- 
pression on  the  different  image  samples: 


f = (E/X)  (b/8) 


E/X 
o o 


where  b is  bits/pel  and  E and  X are  edge  and  threshold  average  descriptors  of 

o o 


original . 


In  almost  all  cases,  f decreased  with  increasing  compression.  Looking  at  the 
images  (Figures  6-7  through  6-19),  one  can  see  deterioration  increasing,  with  the 
compression  level  further  validating  the  values  of  f.  This  relationship  holds  for 
the  test  pattern  as  well  as  each  of  the  FLIR  images.  If  f = 0.5  were  arbitrarily 
selected  for  a cut-off  value,  a data  rate  of  one  bit  per  pel  would  be  obtainable, 
using  the  N-coef f icient  algorithm. 
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7.4  CONCLUSIONS  AND  RECOMMENDATIONS 


The  image  descriptors  can  provide  useful  measures  of  image  change  caused  by  proces- 
sing. For  example,  the  enhancement  experiments  have  shown  that  edge-to-texture 
ratios  increase  when  the  edges  of  an  image  are  enhanced.  Also,  when  an  image  is 
compressed,  a quality  index,  based  on  the  ID's,  can  give  an  index  of  the  compression 
algorithm's  ability  to  conserve  edges  at  the  expense  of  texture  information.  The 
edge-texture  division  is  relatively  insensitive,  however,  to  an  image  state  of 
focus.  The  autofocus  experiments  demonstrated  that  activity  in  general,  rather 
than  just  texture  or  edge,  is  reduced  by  mildly  defocusing  an  image.  Blurs  of  a 
large,  spatial  extent  must  be  examined  to  determine  the  effects  of  focus  on  the  ID's. 

The  contribution  of  each  Haar  term  to  the  total  image  description  must  be  weighted 
to  better  match  the  system  MTF.  When  the  uniform  coefficient  weights  currently 
defined  are  used,  system  noise  can  be  incorrectly  characterized  as  significant 
edge  information.  By  assigning  weights  that  minimize  the  effect  of  noise  on  the 
descriptors,  a more  realistic  image  descriptor  may  be  obtained. 


SECTION  8 

SENSOR  PROCESSOR  UNIT  - DEMONSTRATION  PROTOTYPE 


The  image-processing  algorithms  developed  under  this  contract  and  Northrop' s IR&D 
projects  form  the  basis  for  the  development  of  a test  and  demonstration  prototype 
Sensor  Processor  Unit  (SPU)  and  for  applying  these  concepts  to  the  present  U.S. 

Army  Common  Module  FLIR  as  configured  for  the  Advanced  Attack  Helicopter  (AAH). 
Northrop  was  selected  by  the  U.S.  Army  Development  and  Readiness  Command  for  devel- 
opment of  the  Target  Acquisition/Designation  System  and  a Pilot  Night  Vision  System 
(TADS/PNVS)  for  the  AAH.  The  TADS/PNVS  uses  the  common  module  FLIR.  Therefore, 
a demonstration  "brassboard"  of  a FLIR  Sensor  Processor  Unit  would  be  both  timely 
and  pertinent  to  the  Army's  Advanced  Attack  Helicopter  Weapon  System. 

8.1  SYSTEM  DESCRIPTION,  SENSOR  PROCESSING  UNIT 


The  SPU  will  serve  as  an  adjunct  to  the  Common-Module  FLIR  (MODFLIR) . Its  primary 
objectives  include  automatic  image  enhancement  and  multiple-target  detection  and 
classification. 

To  this  end,  specific  design  goals  have  been  enumerated.  They  include  dc  restora- 
tion, local  area  gain  and  brightness  control,  edge  enhancement,  target  detection, 
target  classification,  and  cueing.  These,  in  turn,  have  prompted  several  important 
design  considerations:  interfacing  with  the  FLIR,  real-time  operating  capabilities 
such  as  automatic  focus,  freedom  from  artifacts,  and  acceptable  probabilities  of 
detection,  classification,  and  cueing. 

In  terms  of  major  operations,  the  SPU  first  seeks  to  enhance  the  image  before  inter- 
rogating it.  Effectively,  enhancement  increases  accessibility  of  image  information 
before  actually  using  it.  To  this  end,  it  employs  Haar  image  transforms.  In  gen- 
eral, Haar  transforms  simplify  the  approach  to  image  enhancement  by  dividing  the 
picture  into  global  and  local  functions  that  somewhat  resemble  desired  structures 
in  the  enhanced  scene.  Thus,  dc  restoration  uses  multiple  dc  Haar  functions  from 
a horizontal  one-dimensional  (1-D)  transform,  brightness  control  employs  the  single 
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dc  Haar  from  a horizontal  and  vertical  two-dimensional  (2-D)  transform,  while  gain 
control  and  auto  focus  use  all  Haar  terms  but  dc.  Haar  transforms  are  used  to 
an  even  greater  extent  in  target  detection,  where  the  ability  of  local  functions 
to  represent  restricted  regions  of  the  image  while  ignoring  other  areas  becomes 
quite  important. 

In  target  detection,  the  object  is  to  apply  multiple  algorithms  that  discriminate 
in  favor  of  likely  points  or  areas  of  interest.  These  algorithms  make  use  of  high 
relative  temperature,  sharp  corners,  and  straight  edges  to  distinguish  man-made 
targets  from  a nominally  natural  background.  Their  combined  use  helps  to  increase 
probability  of  detection  while  decreasing  probability  of  false  alarm.  Weighted 
sums  of  individual  detector  outputs  will  be  compared  to  a preset  threshold  level. 


Target  classification  operates  only  on  objects  selected  during  the  detection  phase. 
Relatively  simple  operations  on  this  transform  enable  size,  translation,  and  rotation 
invariance  in  distinguishing  different  targets.  Target-recognition  output  overlays 
appropriate  display  cues  by  means  of  the  flag-symbol  generator  and  scan  converter. 

The  output  also  feeds  back  within  the  function  to  minimize  occurrence  of  false 
alarms,  and  to  the  target-detection  function  to  enable  a self-constructed  library. 

In  addition,  the  target-detection  output  feeds  back  to  image  enhancement  as  a further 
means  of  keeping  false  alarms  to  a minimum. 

8.2  SYSTEM  BLOCK  DIAGRAM 

The  basic  system  block  diagram  appears  in  Figure  8-1.  Image  data  from  the  MODFLIR 
is  digitized  and  presented  in  a rectangular  row-column  format.  Each  row  is  then 
Haar-transformed . 

At  this  point,  dc  restoration  is  applied.  This  operation  corrects  the  degradation 
caused  by  the  ac  coupling  of  each  photodetector  in  the  parallel-scan  FLIR. 

A vertical  1-D  Haar  transformation  then  completes  the  2-D  Haar  matrix.  This  opera- 
tion must  simultaneously  draw  on  information  from  consecutive  pairs  of  interlaced 
sweeps  of  the  race-track  scan  and  uses  a field  store  for  this  purpose. 


< 
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Figure  8-1.  TOP-LEVEL  FUNCTIONAL  BLOCK  DIAGRAM 

Local  area  gain  and  brightness  control  now  takes  place  in  the  2-D  Haar  domain. 
Brightness  is  controlled  by  the  dc  Haar  term  while  gain  (contrast)  is  regulated 
by  the  remaining  terms.  Edge-extraction,  edge-enhancement  and/or  bandwidth- 
reduction  is  then  applied.  These  operations  apply  to  each  local  area,  or  16  x 
16-pel  sector,  into  which  the  overall  frame  has  been  partitioned.  A special  averag- 
ing algorithm  is  applied  to  reduce  artifacts.  Here,  servo  command  signals  for 
automatic  focus  of  the  FLIR  lens  system  are  also  generated. 

The  improved  2-D  Haar  data  is  subsequently  converted  back  to  the  image  domain  via 
the  inverse  Haar  transform.  It  will  produce  an  image  that  is  free  of  dc-restoration- 
type  problems.  The  resulting  image  will  further  show  moderated  responses  to  bright- 
ness changes  and  scene  contrast  variations  that  are  designed  for  optimum  use  of 
the  available  display  gray  scale.  A scan  converter  changes  the  image  information 
to  a 525-line  serial  format.  Appropriate  target-designation  flags  are  superimposed 
and  the  result  converted  to  an  analog  TV-type  signal. 


A second  output  from  the  image  enhancement  functions  simultaneously  activates 
the  detection  and  classification  phases  of  the  system.  Detection  incorporates 
separate  algorithms  for  sensing  hotspot,  corner,  edge  and  other  characteristics 
of  a possible  target. 

Output  from  the  target  detection  operation  consists  of  coordinate  location  of 
available  targets  in  the  displayed  scene.  This  effectively  calls  out  the  immedi- 
ate locale  of  each  available  target  as  an  area  requiring  further  analysis.  A 
pattern  limiter  and  normalizer  expands  each  original  target  spot  and  its  locale 
to  a standard  scale  size  and  limits  the  displayed  target  intensity.  The  processed 
image  is  then  2-D  Fourier  transformed.  Absolute  values  of  the  transform  are  inte- 
grated to  provide  a translation  and  rotation  invariant  numerical  indicator  of  tar- 
get class.  Specific  numbers  or  groups  of  numbers  then  trigger  appropriate  flags 
in  the  symbol  generator.  Classification  information  can  be  fed  back  to  the  detec- 
tion, storage,  and  limiter-normalizer  stages  to  improve  their  operations. 

The  overall  system,  therefore,  presents  an  enhanced  FLIR  image  with  appropriate 
flags  designating  and  classifying  particular  targets  within  the  scene.  Different 
options,  such  as  target  motion  detection  with  a Moving  Target  Indicator  (MTI)  and 
multiple  limiters,  and  their  practical  effects  in  reducing  false  alarms  and  improv- 
ing detection  will  be  studied  further. 

The  proposed  hardware  implementation  involves  maximum  use  of  Large  Scale  Integra- 
tion (LSI).  High-speed  microprocessor  modules  have  been  selected  that  offer  a 
powerful  capability  for  flexibility  and  growth.  This  technology  is  an  extension 
of  many  years  of  design,  fabrication,  and  testing  of  microprocessor  systems. 

8.3  HARDWARE  CONFIGURATION 


The  sensor  processor  unit  (Figure  8-2)  will  consist  of  an  enclosure,  five  17-  by 
7-inch  circuit  cards,  cooling  fans,  and  a wiring  harness. 

Although  this  unit  will  not  be  built  to  meet  military  specifications,  it  will  be 
built  to  withstand  flight  test  and  expected  environmental  extremes.  Best  commer- 
cial practices  will  be  used  throughout. 


8-4 


ADDENDUM 

AUTOFOCUS  DEMONSTRATION 


A- 1 . INTRODUCTION 


In  Section  2 (Autofocus  Experiments)  a method  of  automatically  focusing  an 
imaging  system  was  developed.  The  procedure  developed  for  detecting  and  cor- 
recting the  focus  condition  of  an  image  consists  of  measuring  the  total  power  of 
the  ac  Haar  coefficients  of  the  image  while  controlling  the  lens  focus  adjustment 
by  electromechanical  means.  As  the  lens  is  defocused  (in  either  direction  from 
optimum  focus  position),  the  observed  sequency  power  decreases.  This  results  in 
an  error  signal  determining  the  optimum  focus  position. 

In  order  to  fully  evaluate  this  method  of  autofocus,  an  autofocus  unit  was  con- 
structed and  interfaced  with  the  two-dimensional  Haar  sensor  processor.  The  auto- 
focus system  was  then  demonstrated  at  Wright  Patterson  Air  Force  Base  during 
September  1977  using  a Honeywell  Chaparral  Forward  Looking  Infrared  (FLIR).  The 
FLIR  and  autofocus  system  were  located  in  a tower  overlooking  the  base  and  beyond. 
Tests  were  run  at  different  hours  of  the  day  and  at  night,  under  varied  thermal 
radiating  conditions.  The  details  of  the  autofocus  system  and  the  results  of  the 
demonstration  are  presented  in  this  addendum. 


A-2 . SYSTEM  CONFIGURATION 


The  overall  configuration  of  the  autofocus  system  is  presented  in  Figure  A-l. 
Image  information  is  output  from  the  FLIR  as  standard  RS-170  video.  The  two- 
dimensional  Haar  sensor  processor  digitizes  the  video  and  computes  the  two- 
dimensional  Haar  transform  of  each  16  x 16  pel  sector  in  each  field  of  the 
digitized  image.  Both  the  digitized  and  transformed  video  are  then  routed  to  the 
inverse  Haar  processor  where  the  transformed  image  is  reconstructed  and  displayed 
along  with  the  digitized  image.  Transformed  video  is  also  accepted  by  the  auto- 
focus unit.  Here,  a measure  of  the  image's  state  of  focus  is  evaluated  and  lens 
servo  commands  are  originated. 
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Figure  A-l.  AUTOFOCUS  SYSTEM  CONFIGURATION 


A more  detailed  system  diagram  of  the  autofocus  unit  is  given  in  Figure  A-2.  The 
transformed  video,  in  the  form  of  Haar  coefficients,  is  first  input  to  a hardware 
multiply  circuit  where  all  the  ac  coefficients  are  squared.  The  square  ac  terms 
are  then  summed  over  the  entire  sector.  The  number  and  location  of  the  sectors 
to  be  summed  in  this  fashion  are  determined  by  front  panel  control  settings  on 
the  autofocus  unit,  see  Figure  A-3  (any  location  on  the  video  monitor  and  up  to 
15  sectors  may  be  selected).  After  the  ac  power  is  summed  over  a number  of  sectors, 
it  is  then  integrated  over  a number  of  fields.  The  amount  of  temporal,  or  field,  • 
integration  is  also  set  on  the  front  panel  (up  to  15  fields  may  be  selected). 

The  output  of  the  field  memory  is  the  focus  measure. 
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The  focus  measure  is  both  displayed  and  used  as  input  to  the  servo  control  func- 
tion. The  servo  control  in  turn  focuses  the  system  by  continually  moving  the  lens 
by  small  increments  in  an  effort  to  maximize  the  focus  measure.  Although  the  sys- 
tem will  never  come  to  rest  precisely  at  the  optimum  focus  position  utilizing  this 
approach,  the  lens  excursions  about  that  position  may  be  small  enough  to  present 
no  visible  degradation  in  image  quality. 
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Figure  A-2.  AUTOFOCUS  UNIT  BLOCK  DIAGRAM 
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The  servo  control  may  seek  out  the  optimum  focus  position  in  several  ways.  A 
method  that  keeps  lens  excursions  about  the  optimum  position  to  a minimum  is 
depicted  in  Figure  A-4.  Small  oscillations  are  first  induced  in  the  focus  adjust- 
ment by  the  servo  control.  The  phase  of  the  corresponding  oscillations  in  the 
focus  measure  then  indicates  which  side  of  the  focus  curve  the  lens  is  on.  The 
focus  is  adjusted  towards  optimum  and  the  phase  is  reevaluated  for  the  next  step. 
When  the  optimum  focus  position  is  reached,  a second  harmonic  in  the  focus  measure 
appears . 
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Although  this  method  of  focus  control  is  efficient  in  terms  of  noise  rejection 
and  economy  of  lens  travel,  it  places  several  restrictions  on  the  focus  servo 
system.  For  example,  since  small  rapid  oscillations  of  a lens  are  required,  the 
mass  of  the  focusing  lens  is  limited.  Also,  backlash  of  the  focus  gear  train  may 
introduce  large  nonlinearities  into  the  focus  control  system. 

A more  generally  applicable  method  of  seeking  the  optimum  focus  position  is  to 
move  the  focus  adjustment  in  one  direction  at  a uniform  rate  and,  if  a downward 
trend  in  the  focus  measure  is  observed,  change  directions.  The  system  will  there- 
fore be  constantly  dithering  across  the  optimum  focus  position.  This  approach 
moves  the  lens  slightly  farther  from  the  optimum  position  than  the  previous 
method.  By  using  small  focus  adjustment  increments,  however,  it  is  still  possible 
to  keep  any  degradation  in  image  quality  below  an  observable  level.  Previous 
autofocus  experiments  indicate  that  a 15  percent  reduction  in  the  focus  measure 
cannot  be  detected  on  the  video  monitor. 

A degree  of  noise  rejection  is  achieved  by  controlling  the  system's  sensitivity 
to  changes  in  the  focus  measure.  The  servo  control  logic  will  reverse  the  direc- 
tion of  lens  travel  only  if  three  consecutive  decreases  in  focus  measure  are  ex- 
perienced and  the  decreases  are  of  sufficient  magnitude.  The  thresholding  of  the 
amount  of  focus  measure  degradation  is  performed  by  the  sensitivity  control  shown 
in  Figure  A-2.  The  sensitivity  control  functions  by  adding  a fixed  threshold  value 
to  each  incoming  focus  measure.  The  amount  of  focus  measure  degradation  must  then 
exceed  the  threshold  in  order  to  be  indicated  as  a negative  slope  by  the  slope 
detector.  This  method  of  reversing  direction  on  focus  degradation  was  used  for 
the  autofocus  demonstration.  This  was  necessary  since  the  lens  servo  mechanism 
of  the  FLIR  consisted  of  an  unspecified  mass  driven  by  an  unspecified  motor.  Thf 
gear  train  connecting  the  motor  and  its  optical  element  also  exhibited  a 4 percent 
of  full-scale  backlash. 

A- 3.  SYSTEM  DATA 

As  a preliminary  to  the  automated  operation  of  the  focusing  system,  static  data 
was  collected  to  evaluate  the  focus  measure  as  a function  of  lens  position.  This 
was  done  to  validate  the  laboratory  concept  of  autofocus  and  to  determine  the  system 
parameters  relevant  to  the  automatic  operation  of  the  system.  Such  system  parameters 
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include:  1)  noise  levels  in  the  focus  measure,  2)  sensitivity  and  scaling  require- 

ments to  detect  meaningful  changes  in  the  focus  measure,  3)  uncertainty  of  the 
lens  position,  and  4)  the  desired  lens  position  increment  size. 

Two  image  areas  chosen  for  consideration  are  flagged  by  reduced  brightness  in  Fig- 
ure A-5.  They  consist  of  a radio  tower  in  the  foreground  (approximate  range  50 
meters)  and  garage  in  the  background  (approximate  range  300  meters).  The  background 
focus  scene  is  focused  at  about  20  percent  of  full-scale  adjustment  away  from  the 
infinity  setting.  The  degree  of  focus  difference  between  these  two  ranges  may 
be  seen  by  observing  the  portion  of  defocused  radio  tower  in  the  background  focus 
scene . 

Focus  measure  curves  derived  from  the  two  image  areas  are  shown  in  Figures  A-6  and 
A-7.  The  focus  measure  curves  were  generated  by  manually  increasing  or  decreasing 
the  lens  adjustment  servo  command  in  discrete  steps,  and  observing  a sample  of  the 
focus  measure  at  each  step.  As  a starting  point,  the  step  size  was  set  such  that 
100  steps  were  required  for  full-scale  focus  adjustment.  The  focus  measure  was 
integrated  over  four  sectors  of  the  image  and  two  frames  of  video.  The  correspond- 
ing update  rate  of  the  focus  measure  is  15  Hz.  Two  sets  of  data,  increasing  and 
decreasing  focus  adjustment  command,  are  presented  for  each  image  segment.  This 
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Figure  A-5.  FOCUS  AREAS 


FOCUS  MEASURE  FOCUS  MEASURE 

MAXIMUM  FOCUS  MEASURE  MAXIMUM  FOCUS  MEASURE 


H 

was  done  Co  illustrate  the  degree  of  backlash  in  the  focus  gear  train.  As  shown 
in  Figures  A-6  and  A-7,  the  focus  measures  resulting  from  increasing  focus  adjustment 
commands  exhibit  a peak  well  to  the  right  of  the  curves  resulting  from  decreasing 
focus  adjustment.  The  steepness  of  the  focus  curve  from  the  foreground  target 
(Figure  A-6)  required  the  focus  adjustment  step  size  to  be  reduced  by  a factor 
of  two. 

Although  both  sets  of  curves  show  peaks  at  the  focus  point,  the  curves  due  to  the 
background  scene  (Figure  A-7)  exhibits  a much  lower  signal-to-noise  ratio  than 
the  foreground  scene.  This  is  due  to  the  reduced  image  contrast  and,  hence,  lower 
focus  measure  in  the  background  scene.  The  signal-to-noise  ratio  in  the  background 
scene  is  14  dB  while  the  foreground  exhibits  a 20  dB  signal-to-noise  ratio. 


Although  the  signal-to-noise  level  observed  in  the  static  data  is  somewhat  low, 
the  system  still  dithers  about  the  optimum  focus  position  when  operating  in  the 
automatic  mode.  The  limits  of  focus  dither  coresponding  to  maximum  focus  measure 
sensitivity  for  both  scenes  are  given  in  Table  A-l.  A lens  travel  of  about  8 posi- 
tions was  experienced  in  each  case.  Although  the  focus  adjustment  varied  by  these 
amounts,  the  image's  subjective  state  of  focus  remained  constant.  By  using  the 


median  value  of  the  focus  adjustment  as  the  focus  point  and  calibrating  the  focus 
adjustment  scale  in  units  of  focus  distance,  the  autofocus  system  can  then  function 


as  a passive  monocular  autoranging  device. 


Table  A-l.  LIMITS  OF  FOCUS  ADJUSTMENT  IN  AUTOMATIC  MODE 


The  step  response  of  the  autofocus  system  was  measured  by  focusing  on  the  background 
scene  and  changing  the  area  of  consideration  for  focus  to  the  foreground  target. 

It  was  observed  that  an  average  of  12  seconds  was  required  for  the  system  to  identify 
the  foreground  focus  position.  The  system  step  response  in  this  case  is  limited 
by  the  lens  adjustment  step  rate.  The  system,  as  configured  for  the  demonstration, 
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adjusts  the  lens  in  half-step  increments  once  every  350  milliseconds.  This  is 
done  to  minimize  the  workload  of  the  lens  servo.  However,  since  the  focus  measure 
is  being  updated  at  15  Hz,  a 5 to  1 improvement  in  step  response  could  be  realized 
by  using  a tighter  lens  servo  system. 


A-4.  CONCLUSIONS  AND  RECOMMENDATIONS 


The  autofocus  demonstration  has  shown  the  feasibility  of  an  operational  autofoci 
system.  The  observed  noise  values  of  the  focus  measure  identify  the  need  for  a 
degree  of  noise  filtering  before  the  focus  is  evaluated.  The  speed  of  response 
and  positional  stability  of  the  lens  servo  system  also  requires  significant 
improvement . 


